Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 14:11:54.952630 744 test_util.cc:268] Using random seed: 1296861038
I20240503 14:11:54.964911 744 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:11:54.965013 744 ts_itest-base.cc:113] --------------
I20240503 14:11:54.965075 744 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:11:54.965131 744 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:11:54.965204 744 ts_itest-base.cc:116] --------------
2024-05-03T14:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:11:54Z Disabled control of system clock
I20240503 14:11:54.991464 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.186.62:45027
--webserver_interface=127.0.186.62
--webserver_port=0
--builtin_ntp_servers=127.0.186.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.186.62:45027
--enable_range_replica_placement=false with env {}
W20240503 14:11:55.321991 756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:55.322356 756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:55.322507 756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:55.330153 756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:11:55.330305 756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:55.330394 756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:11:55.330478 756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:11:55.339833 756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.186.62:45027
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.186.62:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.186.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:55.341411 756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:55.343353 756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:55.355670 761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.356415 762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.356943 764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.358235 756 server_base.cc:1034] running on GCE node
I20240503 14:11:55.359088 756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:55.360697 756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:55.361984 756 hybrid_clock.cc:648] HybridClock initialized: now 1714745515361917 us; error 43 us; skew 500 ppm
I20240503 14:11:55.362329 756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.364749 756 webserver.cc:460] Webserver started at http://127.0.186.62:35679/ using document root <none> and password file <none>
I20240503 14:11:55.365438 756 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.365557 756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.365825 756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.368692 756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b271483fbc2940dda9d0fc582034b140"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-mh2c"
I20240503 14:11:55.369431 756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b271483fbc2940dda9d0fc582034b140"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-mh2c"
I20240503 14:11:55.374825 756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:11:55.378417 770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.379809 756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:55.379976 756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b271483fbc2940dda9d0fc582034b140"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-mh2c"
I20240503 14:11:55.380167 756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:55.408354 756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:55.409224 756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:55.409541 756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:55.428310 756 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.62:45027
I20240503 14:11:55.428329 821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.62:45027 every 8 connection(s)
I20240503 14:11:55.429796 756 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:11:55.433681 822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:55.439842 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 756
I20240503 14:11:55.440164 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:11:55.445190 822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140: Bootstrap starting.
I20240503 14:11:55.449062 822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:55.450202 822 log.cc:824] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:55.453074 822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140: No bootstrap required, opened a new log
I20240503 14:11:55.460687 822 raft_consensus.cc:348] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b271483fbc2940dda9d0fc582034b140" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45027 } }
I20240503 14:11:55.461238 822 raft_consensus.cc:374] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:55.461390 822 raft_consensus.cc:729] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b271483fbc2940dda9d0fc582034b140, State: Initialized, Role: FOLLOWER
I20240503 14:11:55.462217 822 consensus_queue.cc:260] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b271483fbc2940dda9d0fc582034b140" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45027 } }
I20240503 14:11:55.462484 822 raft_consensus.cc:388] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:11:55.462647 822 raft_consensus.cc:482] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:11:55.462853 822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:55.466214 822 raft_consensus.cc:504] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b271483fbc2940dda9d0fc582034b140" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45027 } }
I20240503 14:11:55.466856 822 leader_election.cc:302] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b271483fbc2940dda9d0fc582034b140; no voters:
I20240503 14:11:55.467576 822 leader_election.cc:288] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:11:55.467773 827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:55.469183 827 raft_consensus.cc:686] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [term 1 LEADER]: Becoming Leader. State: Replica: b271483fbc2940dda9d0fc582034b140, State: Running, Role: LEADER
I20240503 14:11:55.469735 827 consensus_queue.cc:237] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b271483fbc2940dda9d0fc582034b140" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45027 } }
I20240503 14:11:55.469911 822 sys_catalog.cc:564] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:11:55.472771 829 sys_catalog.cc:455] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b271483fbc2940dda9d0fc582034b140. Latest consensus state: current_term: 1 leader_uuid: "b271483fbc2940dda9d0fc582034b140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b271483fbc2940dda9d0fc582034b140" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45027 } } }
I20240503 14:11:55.473116 829 sys_catalog.cc:458] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:55.473668 828 sys_catalog.cc:455] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b271483fbc2940dda9d0fc582034b140" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b271483fbc2940dda9d0fc582034b140" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45027 } } }
I20240503 14:11:55.474031 828 sys_catalog.cc:458] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140 [sys.catalog]: This master's current role is: LEADER
I20240503 14:11:55.476852 833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:11:55.482637 833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:11:55.491068 833 catalog_manager.cc:1347] Generated new cluster ID: 0c58ecc534ea48fb9595037b83c77928
I20240503 14:11:55.491209 833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:11:55.533891 833 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:11:55.536005 833 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:11:55.557323 833 catalog_manager.cc:5857] T 00000000000000000000000000000000 P b271483fbc2940dda9d0fc582034b140: Generated new TSK 0
I20240503 14:11:55.558321 833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:11:55.577168 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.1:0
--local_ip_for_outbound_sockets=127.0.186.1
--webserver_interface=127.0.186.1
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--builtin_ntp_servers=127.0.186.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 14:11:55.920692 846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:55.921017 846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:55.921116 846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:55.928711 846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:55.928906 846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.1
I20240503 14:11:55.938035 846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.186.1
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.1
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:55.939424 846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:55.941296 846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:55.955055 851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.955628 852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:55.956126 854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:55.957144 846 server_base.cc:1034] running on GCE node
I20240503 14:11:55.957924 846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:55.959002 846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:55.960268 846 hybrid_clock.cc:648] HybridClock initialized: now 1714745515960195 us; error 45 us; skew 500 ppm
I20240503 14:11:55.960613 846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:55.963121 846 webserver.cc:460] Webserver started at http://127.0.186.1:44343/ using document root <none> and password file <none>
I20240503 14:11:55.963946 846 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:55.964061 846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:55.964340 846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:55.967020 846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0e54d3bc4e404b909a2fb47cf980770e"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-mh2c"
I20240503 14:11:55.967911 846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0e54d3bc4e404b909a2fb47cf980770e"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-mh2c"
I20240503 14:11:55.973672 846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:11:55.977447 860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:55.979161 846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:11:55.979365 846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0e54d3bc4e404b909a2fb47cf980770e"
format_stamp: "Formatted at 2024-05-03 14:11:55 on dist-test-slave-mh2c"
I20240503 14:11:55.979689 846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.001369 846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.002244 846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.002533 846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.003494 846 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:56.005070 846 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:56.005151 846 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.005290 846 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:56.005345 846 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.037190 846 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.1:42633
I20240503 14:11:56.037245 972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.1:42633 every 8 connection(s)
I20240503 14:11:56.038594 846 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:11:56.045620 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 846
I20240503 14:11:56.045826 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:11:56.049082 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.2:0
--local_ip_for_outbound_sockets=127.0.186.2
--webserver_interface=127.0.186.2
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--builtin_ntp_servers=127.0.186.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 14:11:56.052758 973 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45027
I20240503 14:11:56.053210 973 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:56.054170 973 heartbeater.cc:511] Master 127.0.186.62:45027 requested a full tablet report, sending...
I20240503 14:11:56.056821 787 ts_manager.cc:194] Registered new tserver with Master: 0e54d3bc4e404b909a2fb47cf980770e (127.0.186.1:42633)
I20240503 14:11:56.059186 787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.1:36235
W20240503 14:11:56.384392 977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.384737 977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.384896 977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.392488 977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.392716 977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.2
I20240503 14:11:56.403048 977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.186.2
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.2
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.404559 977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.406319 977 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.419190 982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.419767 985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.420729 983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.421342 977 server_base.cc:1034] running on GCE node
I20240503 14:11:56.421973 977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.422973 977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.424247 977 hybrid_clock.cc:648] HybridClock initialized: now 1714745516424192 us; error 30 us; skew 500 ppm
I20240503 14:11:56.424587 977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.426991 977 webserver.cc:460] Webserver started at http://127.0.186.2:35451/ using document root <none> and password file <none>
I20240503 14:11:56.427819 977 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.427970 977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.428280 977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.430871 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1b5ad59a974e48c78e85ff29a545dc7e"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-mh2c"
I20240503 14:11:56.431747 977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1b5ad59a974e48c78e85ff29a545dc7e"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-mh2c"
I20240503 14:11:56.437554 977 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:11:56.441205 991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.442713 977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:11:56.442889 977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1b5ad59a974e48c78e85ff29a545dc7e"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-mh2c"
I20240503 14:11:56.443073 977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.465565 977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.466472 977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.466769 977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.467859 977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:56.469475 977 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:56.469552 977 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.469688 977 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:56.469759 977 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.502014 977 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.2:38595
I20240503 14:11:56.502048 1103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.2:38595 every 8 connection(s)
I20240503 14:11:56.503342 977 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:11:56.508867 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 977
I20240503 14:11:56.509057 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:11:56.512466 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.3:0
--local_ip_for_outbound_sockets=127.0.186.3
--webserver_interface=127.0.186.3
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--builtin_ntp_servers=127.0.186.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 14:11:56.518134 1104 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45027
I20240503 14:11:56.518502 1104 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:56.519374 1104 heartbeater.cc:511] Master 127.0.186.62:45027 requested a full tablet report, sending...
I20240503 14:11:56.520977 787 ts_manager.cc:194] Registered new tserver with Master: 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2:38595)
I20240503 14:11:56.521773 787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.2:57365
W20240503 14:11:56.853482 1107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:56.853829 1107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:56.853992 1107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:56.861869 1107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:56.862112 1107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.3
I20240503 14:11:56.871286 1107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.186.3
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.3
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:56.872767 1107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:56.874738 1107 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:56.887136 1112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.887998 1115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:56.888559 1113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:56.889303 1107 server_base.cc:1034] running on GCE node
I20240503 14:11:56.889948 1107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:56.890937 1107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:56.892154 1107 hybrid_clock.cc:648] HybridClock initialized: now 1714745516892091 us; error 45 us; skew 500 ppm
I20240503 14:11:56.892467 1107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:56.894872 1107 webserver.cc:460] Webserver started at http://127.0.186.3:34885/ using document root <none> and password file <none>
I20240503 14:11:56.895679 1107 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:56.895815 1107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:56.896121 1107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:56.898784 1107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8829c54b55bc42babd683117bf1a1f59"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-mh2c"
I20240503 14:11:56.899565 1107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8829c54b55bc42babd683117bf1a1f59"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-mh2c"
I20240503 14:11:56.907788 1107 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20240503 14:11:56.911312 1121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.912778 1107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:56.912947 1107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8829c54b55bc42babd683117bf1a1f59"
format_stamp: "Formatted at 2024-05-03 14:11:56 on dist-test-slave-mh2c"
I20240503 14:11:56.913133 1107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:56.933245 1107 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:56.934106 1107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:56.934407 1107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:56.935374 1107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:56.936930 1107 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:56.937016 1107 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.937147 1107 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:56.937203 1107 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:56.968993 1107 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.3:36485
I20240503 14:11:56.969026 1233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.3:36485 every 8 connection(s)
I20240503 14:11:56.970355 1107 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:11:56.971863 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 1107
I20240503 14:11:56.972059 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:11:56.975317 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.4:0
--local_ip_for_outbound_sockets=127.0.186.4
--webserver_interface=127.0.186.4
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--builtin_ntp_servers=127.0.186.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 14:11:56.985400 1234 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45027
I20240503 14:11:56.985776 1234 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:56.986764 1234 heartbeater.cc:511] Master 127.0.186.62:45027 requested a full tablet report, sending...
I20240503 14:11:56.988343 787 ts_manager.cc:194] Registered new tserver with Master: 8829c54b55bc42babd683117bf1a1f59 (127.0.186.3:36485)
I20240503 14:11:56.989126 787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.3:38235
I20240503 14:11:57.062726 973 heartbeater.cc:503] Master 127.0.186.62:45027 was elected leader, sending a full tablet report...
W20240503 14:11:57.319376 1237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.319754 1237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.319913 1237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.327488 1237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.327761 1237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.4
I20240503 14:11:57.336987 1237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.186.4
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.4
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.338443 1237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.340322 1237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.353338 1242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.354106 1243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.354928 1245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.355865 1237 server_base.cc:1034] running on GCE node
I20240503 14:11:57.356546 1237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.357523 1237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.358747 1237 hybrid_clock.cc:648] HybridClock initialized: now 1714745517358668 us; error 56 us; skew 500 ppm
I20240503 14:11:57.359048 1237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.361515 1237 webserver.cc:460] Webserver started at http://127.0.186.4:40987/ using document root <none> and password file <none>
I20240503 14:11:57.362269 1237 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.362409 1237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.362712 1237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.365480 1237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "f270bf4de970457dadd6b9f34c70fb05"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-mh2c"
I20240503 14:11:57.366272 1237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "f270bf4de970457dadd6b9f34c70fb05"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-mh2c"
I20240503 14:11:57.371562 1237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:11:57.375010 1251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.376363 1237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:11:57.376518 1237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "f270bf4de970457dadd6b9f34c70fb05"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-mh2c"
I20240503 14:11:57.376693 1237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.397243 1237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.398115 1237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.398417 1237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.399385 1237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.401134 1237 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.401268 1237 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240503 14:11:57.401412 1237 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.401474 1237 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.438210 1237 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.4:37377
I20240503 14:11:57.438292 1363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.4:37377 every 8 connection(s)
I20240503 14:11:57.439581 1237 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:11:57.446384 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 1237
I20240503 14:11:57.446590 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:11:57.450057 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.5:0
--local_ip_for_outbound_sockets=127.0.186.5
--webserver_interface=127.0.186.5
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--builtin_ntp_servers=127.0.186.20:35715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 14:11:57.453583 1364 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45027
I20240503 14:11:57.453923 1364 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.454833 1364 heartbeater.cc:511] Master 127.0.186.62:45027 requested a full tablet report, sending...
I20240503 14:11:57.456429 787 ts_manager.cc:194] Registered new tserver with Master: f270bf4de970457dadd6b9f34c70fb05 (127.0.186.4:37377)
I20240503 14:11:57.457653 787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.4:43179
I20240503 14:11:57.524515 1104 heartbeater.cc:503] Master 127.0.186.62:45027 was elected leader, sending a full tablet report...
W20240503 14:11:57.793501 1367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:11:57.793812 1367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:11:57.793910 1367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:11:57.801537 1367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:11:57.801735 1367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.5
I20240503 14:11:57.810851 1367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:35715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.186.5
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45027
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.5
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:11:57.812276 1367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:11:57.814057 1367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:11:57.827010 1372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.827828 1375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:11:57.828469 1373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:11:57.829247 1367 server_base.cc:1034] running on GCE node
I20240503 14:11:57.829907 1367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:11:57.830917 1367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:11:57.832134 1367 hybrid_clock.cc:648] HybridClock initialized: now 1714745517832057 us; error 62 us; skew 500 ppm
I20240503 14:11:57.832444 1367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:11:57.839359 1367 webserver.cc:460] Webserver started at http://127.0.186.5:35103/ using document root <none> and password file <none>
I20240503 14:11:57.840240 1367 fs_manager.cc:359] Metadata directory not provided
I20240503 14:11:57.840394 1367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:11:57.840718 1367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:11:57.843438 1367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-mh2c"
I20240503 14:11:57.844362 1367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-mh2c"
I20240503 14:11:57.850157 1367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:11:57.854110 1381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.855674 1367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:11:57.855849 1367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8"
format_stamp: "Formatted at 2024-05-03 14:11:57 on dist-test-slave-mh2c"
I20240503 14:11:57.856050 1367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:11:57.876878 1367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:11:57.877784 1367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:11:57.878064 1367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:11:57.879045 1367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:11:57.880560 1367 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:11:57.880638 1367 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.880765 1367 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:11:57.880812 1367 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:11:57.918217 1367 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.5:44587
I20240503 14:11:57.918326 1493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.5:44587 every 8 connection(s)
I20240503 14:11:57.919631 1367 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:11:57.920570 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 1367
I20240503 14:11:57.920768 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:11:57.932495 1494 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45027
I20240503 14:11:57.932837 1494 heartbeater.cc:464] Registering TS with master...
I20240503 14:11:57.933729 1494 heartbeater.cc:511] Master 127.0.186.62:45027 requested a full tablet report, sending...
I20240503 14:11:57.935199 787 ts_manager.cc:194] Registered new tserver with Master: 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5:44587)
I20240503 14:11:57.936131 787 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.5:35083
I20240503 14:11:57.937785 744 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:11:57.959360 787 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33186:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:11:57.991796 1234 heartbeater.cc:503] Master 127.0.186.62:45027 was elected leader, sending a full tablet report...
I20240503 14:11:58.002468 1039 tablet_service.cc:1467] Processing CreateTablet for tablet b3969333edf443b28972d1b094beeb16 (DEFAULT_TABLE table=TestTable [id=fbd7f9226a3545bab7bd2c0434de3d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:58.004263 1039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3969333edf443b28972d1b094beeb16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.006826 908 tablet_service.cc:1467] Processing CreateTablet for tablet b3969333edf443b28972d1b094beeb16 (DEFAULT_TABLE table=TestTable [id=fbd7f9226a3545bab7bd2c0434de3d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:58.006938 1299 tablet_service.cc:1467] Processing CreateTablet for tablet b3969333edf443b28972d1b094beeb16 (DEFAULT_TABLE table=TestTable [id=fbd7f9226a3545bab7bd2c0434de3d51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:11:58.008101 908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3969333edf443b28972d1b094beeb16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.008337 1299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3969333edf443b28972d1b094beeb16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:11:58.019472 1512 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: Bootstrap starting.
I20240503 14:11:58.022655 1514 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05: Bootstrap starting.
I20240503 14:11:58.023072 1512 tablet_bootstrap.cc:654] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.023891 1513 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: Bootstrap starting.
I20240503 14:11:58.026297 1514 tablet_bootstrap.cc:654] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.027279 1513 tablet_bootstrap.cc:654] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: Neither blocks nor log segments found. Creating new log.
I20240503 14:11:58.027467 1512 log.cc:824] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:58.027527 1514 log.cc:824] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:58.029459 1513 log.cc:824] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: Log is configured to *not* fsync() on all Append() calls
I20240503 14:11:58.031354 1512 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: No bootstrap required, opened a new log
I20240503 14:11:58.031356 1514 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05: No bootstrap required, opened a new log
I20240503 14:11:58.031688 1512 ts_tablet_manager.cc:1367] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 14:11:58.031687 1514 ts_tablet_manager.cc:1367] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20240503 14:11:58.032672 1513 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: No bootstrap required, opened a new log
I20240503 14:11:58.032968 1513 ts_tablet_manager.cc:1367] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:11:58.039098 1512 raft_consensus.cc:348] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.039788 1512 raft_consensus.cc:374] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.039953 1512 raft_consensus.cc:729] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e54d3bc4e404b909a2fb47cf980770e, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.040812 1512 consensus_queue.cc:260] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.040891 1514 raft_consensus.cc:348] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.041358 1514 raft_consensus.cc:374] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.041499 1514 raft_consensus.cc:729] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f270bf4de970457dadd6b9f34c70fb05, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.041860 1513 raft_consensus.cc:348] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.042364 1513 raft_consensus.cc:374] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:11:58.042282 1514 consensus_queue.cc:260] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.042515 1513 raft_consensus.cc:729] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b5ad59a974e48c78e85ff29a545dc7e, State: Initialized, Role: FOLLOWER
I20240503 14:11:58.043532 1513 consensus_queue.cc:260] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.049482 1513 ts_tablet_manager.cc:1398] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20240503 14:11:58.056262 1512 ts_tablet_manager.cc:1398] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: Time spent starting tablet: real 0.024s user 0.012s sys 0.012s
I20240503 14:11:58.057063 1520 raft_consensus.cc:482] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:11:58.057555 1520 raft_consensus.cc:504] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.058164 1364 heartbeater.cc:503] Master 127.0.186.62:45027 was elected leader, sending a full tablet report...
I20240503 14:11:58.063239 1520 leader_election.cc:288] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e54d3bc4e404b909a2fb47cf980770e (127.0.186.1:42633), f270bf4de970457dadd6b9f34c70fb05 (127.0.186.4:37377)
I20240503 14:11:58.064709 1514 ts_tablet_manager.cc:1398] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05: Time spent starting tablet: real 0.033s user 0.014s sys 0.012s
I20240503 14:11:58.070344 1319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3969333edf443b28972d1b094beeb16" candidate_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f270bf4de970457dadd6b9f34c70fb05" is_pre_election: true
I20240503 14:11:58.070506 928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3969333edf443b28972d1b094beeb16" candidate_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e54d3bc4e404b909a2fb47cf980770e" is_pre_election: true
I20240503 14:11:58.071064 928 raft_consensus.cc:2454] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5ad59a974e48c78e85ff29a545dc7e in term 0.
I20240503 14:11:58.071065 1319 raft_consensus.cc:2454] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b5ad59a974e48c78e85ff29a545dc7e in term 0.
I20240503 14:11:58.071871 995 leader_election.cc:302] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e54d3bc4e404b909a2fb47cf980770e, 1b5ad59a974e48c78e85ff29a545dc7e; no voters:
I20240503 14:11:58.072402 1520 raft_consensus.cc:2781] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:11:58.072569 1520 raft_consensus.cc:482] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:11:58.072674 1520 raft_consensus.cc:3037] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:58.074767 1520 raft_consensus.cc:504] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.075385 1520 leader_election.cc:288] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [CANDIDATE]: Term 1 election: Requested vote from peers 0e54d3bc4e404b909a2fb47cf980770e (127.0.186.1:42633), f270bf4de970457dadd6b9f34c70fb05 (127.0.186.4:37377)
I20240503 14:11:58.075959 928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3969333edf443b28972d1b094beeb16" candidate_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e54d3bc4e404b909a2fb47cf980770e"
I20240503 14:11:58.075982 1319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3969333edf443b28972d1b094beeb16" candidate_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f270bf4de970457dadd6b9f34c70fb05"
I20240503 14:11:58.076187 928 raft_consensus.cc:3037] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:58.076187 1319 raft_consensus.cc:3037] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:11:58.079427 1319 raft_consensus.cc:2454] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5ad59a974e48c78e85ff29a545dc7e in term 1.
I20240503 14:11:58.079427 928 raft_consensus.cc:2454] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b5ad59a974e48c78e85ff29a545dc7e in term 1.
I20240503 14:11:58.080015 993 leader_election.cc:302] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b5ad59a974e48c78e85ff29a545dc7e, f270bf4de970457dadd6b9f34c70fb05; no voters:
I20240503 14:11:58.080391 1520 raft_consensus.cc:2781] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:11:58.080574 1520 raft_consensus.cc:686] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 1 LEADER]: Becoming Leader. State: Replica: 1b5ad59a974e48c78e85ff29a545dc7e, State: Running, Role: LEADER
I20240503 14:11:58.081099 1520 consensus_queue.cc:237] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:11:58.086246 787 catalog_manager.cc:5484] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2). New cstate: current_term: 1 leader_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } health_report { overall_health: UNKNOWN } } }
I20240503 14:11:58.114917 744 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:11:58.117777 744 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0e54d3bc4e404b909a2fb47cf980770e to finish bootstrapping
I20240503 14:11:58.127113 744 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1b5ad59a974e48c78e85ff29a545dc7e to finish bootstrapping
I20240503 14:11:58.134624 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8829c54b55bc42babd683117bf1a1f59 to finish bootstrapping
I20240503 14:11:58.143867 744 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f270bf4de970457dadd6b9f34c70fb05 to finish bootstrapping
I20240503 14:11:58.152400 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 25c5a6b7b8c44ff49ccdb03221a0dba8 to finish bootstrapping
W20240503 14:11:58.191715 1365 tablet.cc:2369] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:11:58.256264 1105 tablet.cc:2369] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:11:58.291977 974 tablet.cc:2369] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:11:58.616003 1551 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:11:58.627453 1557 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:11:58.653826 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:11:58.675395 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:58.688421 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:58.710304 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:58.714864 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:58.938925 1494 heartbeater.cc:503] Master 127.0.186.62:45027 was elected leader, sending a full tablet report...
I20240503 14:11:59.321326 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.324792 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:11:59.334944 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.336606 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.346853 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.501016 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.516898 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.519301 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.523876 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:11:59.542266 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.702436 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.707288 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:11:59.722486 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.724013 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:11:59.732165 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:00.287806 786 ts_manager.cc:292] Set tserver state for 0e54d3bc4e404b909a2fb47cf980770e to MAINTENANCE_MODE
I20240503 14:12:01.827389 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:01.827469 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:01.842497 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:01.845678 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:01.858641 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.025619 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.035200 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:02.037315 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.038450 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.049054 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.226178 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.232496 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:02.237056 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.238844 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.239094 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:02.317970 1059 consensus_queue.cc:237] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } }
I20240503 14:12:02.320502 1319 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:02.320574 928 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:02.321141 1694 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:02.321576 1804 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:12:02.324949 1694 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } }
I20240503 14:12:02.326005 928 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } }
I20240503 14:12:02.328009 1319 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } }
I20240503 14:12:02.406630 786 catalog_manager.cc:5484] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } }
I20240503 14:12:02.415722 1059 consensus_queue.cc:237] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: true } }
I20240503 14:12:02.418682 928 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:12:02.419611 1694 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:12:02.421736 1319 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:12:02.423048 1818 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:12:02.424096 1818 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: true } } }
I20240503 14:12:02.424855 928 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: true } } }
I20240503 14:12:02.427968 1319 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: true } } }
W20240503 14:12:02.428967 992 consensus_peers.cc:485] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e -> Peer 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5:44587): Couldn't send request to peer 25c5a6b7b8c44ff49ccdb03221a0dba8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3969333edf443b28972d1b094beeb16. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:02.506637 772 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b3969333edf443b28972d1b094beeb16 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 14:12:02.508340 787 catalog_manager.cc:5484] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e reported cstate change: config changed from index 2 to 3, NON_VOTER 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:02.862509 1830 ts_tablet_manager.cc:902] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: Initiating tablet copy from peer 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2:38595)
I20240503 14:12:02.864367 1830 tablet_copy_client.cc:308] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:38595
I20240503 14:12:02.871385 1079 tablet_copy_service.cc:140] P 1b5ad59a974e48c78e85ff29a545dc7e: Received BeginTabletCopySession request for tablet b3969333edf443b28972d1b094beeb16 from peer 25c5a6b7b8c44ff49ccdb03221a0dba8 ({username='slave'} at 127.0.186.5:56781)
I20240503 14:12:02.871703 1079 tablet_copy_service.cc:161] P 1b5ad59a974e48c78e85ff29a545dc7e: Beginning new tablet copy session on tablet b3969333edf443b28972d1b094beeb16 from peer 25c5a6b7b8c44ff49ccdb03221a0dba8 at {username='slave'} at 127.0.186.5:56781: session id = 25c5a6b7b8c44ff49ccdb03221a0dba8-b3969333edf443b28972d1b094beeb16
I20240503 14:12:02.874984 1079 tablet_copy_source_session.cc:205] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:02.879737 1830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3969333edf443b28972d1b094beeb16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:02.890524 1830 tablet_copy_client.cc:783] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:02.890942 1830 tablet_copy_client.cc:647] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:02.893720 1830 tablet_copy_client.cc:523] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:02.911953 1830 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: Bootstrap starting.
I20240503 14:12:02.920228 1830 log.cc:824] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:02.924433 1830 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:02.924935 1830 tablet_bootstrap.cc:492] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: Bootstrap complete.
I20240503 14:12:02.925387 1830 ts_tablet_manager.cc:1367] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20240503 14:12:02.932721 1830 raft_consensus.cc:348] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: true } }
I20240503 14:12:02.933233 1830 raft_consensus.cc:729] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25c5a6b7b8c44ff49ccdb03221a0dba8, State: Initialized, Role: LEARNER
I20240503 14:12:02.933779 1830 consensus_queue.cc:260] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: true } }
I20240503 14:12:02.935353 1830 ts_tablet_manager.cc:1398] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20240503 14:12:02.936555 1079 tablet_copy_service.cc:342] P 1b5ad59a974e48c78e85ff29a545dc7e: Request end of tablet copy session 25c5a6b7b8c44ff49ccdb03221a0dba8-b3969333edf443b28972d1b094beeb16 received from {username='slave'} at 127.0.186.5:56781
I20240503 14:12:02.936750 1079 tablet_copy_service.cc:433] P 1b5ad59a974e48c78e85ff29a545dc7e: ending tablet copy session 25c5a6b7b8c44ff49ccdb03221a0dba8-b3969333edf443b28972d1b094beeb16 on tablet b3969333edf443b28972d1b094beeb16 with peer 25c5a6b7b8c44ff49ccdb03221a0dba8
I20240503 14:12:03.269344 1449 raft_consensus.cc:1203] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240503 14:12:03.769170 1849 raft_consensus.cc:1050] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e: attempting to promote NON_VOTER 25c5a6b7b8c44ff49ccdb03221a0dba8 to VOTER
I20240503 14:12:03.770161 1849 consensus_queue.cc:237] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } }
I20240503 14:12:03.772847 1319 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:03.773283 928 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:03.773540 1849 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:03.773633 1449 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 LEARNER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:12:03.773914 1862 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:03.774282 1861 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:12:03.779189 1862 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } } }
I20240503 14:12:03.780205 1449 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } } }
I20240503 14:12:03.785847 928 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } } }
I20240503 14:12:03.820863 1319 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } } }
I20240503 14:12:03.822058 786 catalog_manager.cc:5484] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e reported cstate change: config changed from index 3 to 4, 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "0e54d3bc4e404b909a2fb47cf980770e" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 42633 } attrs { replace: true } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } } }
I20240503 14:12:03.829170 1059 consensus_queue.cc:237] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } }
I20240503 14:12:03.831027 1449 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 FOLLOWER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:03.831670 1861 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:03.834021 1862 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0e54d3bc4e404b909a2fb47cf980770e (127.0.186.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } } }
I20240503 14:12:03.834723 1449 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0e54d3bc4e404b909a2fb47cf980770e (127.0.186.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } } }
I20240503 14:12:03.921581 772 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b3969333edf443b28972d1b094beeb16 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:12:03.923790 1318 raft_consensus.cc:1261] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Refusing update from remote peer 1b5ad59a974e48c78e85ff29a545dc7e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:03.926307 785 catalog_manager.cc:5484] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e reported cstate change: config changed from index 4 to 5, VOTER 0e54d3bc4e404b909a2fb47cf980770e (127.0.186.1) evicted. New cstate: current_term: 1 leader_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:03.928359 1861 consensus_queue.cc:1035] T b3969333edf443b28972d1b094beeb16 P 1b5ad59a974e48c78e85ff29a545dc7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:03.930184 1318 raft_consensus.cc:2932] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 0e54d3bc4e404b909a2fb47cf980770e (127.0.186.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } } }
I20240503 14:12:03.931327 908 tablet_service.cc:1514] Processing DeleteTablet for tablet b3969333edf443b28972d1b094beeb16 with delete_type TABLET_DATA_TOMBSTONED (TS 0e54d3bc4e404b909a2fb47cf980770e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:35064
I20240503 14:12:03.934933 1878 tablet_replica.cc:330] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: stopping tablet replica
I20240503 14:12:03.935441 1878 raft_consensus.cc:2229] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:03.935990 1878 raft_consensus.cc:2258] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:03.937618 1878 ts_tablet_manager.cc:1875] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:04.029094 1878 ts_tablet_manager.cc:1888] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:12:04.029346 1878 log.cc:1196] T b3969333edf443b28972d1b094beeb16 P 0e54d3bc4e404b909a2fb47cf980770e: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/b3969333edf443b28972d1b094beeb16
I20240503 14:12:04.030370 774 catalog_manager.cc:4830] TS 0e54d3bc4e404b909a2fb47cf980770e (127.0.186.1:42633): tablet b3969333edf443b28972d1b094beeb16 (table TestTable [id=fbd7f9226a3545bab7bd2c0434de3d51]) successfully deleted
I20240503 14:12:04.217468 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:04.220662 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.221046 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.228629 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.244129 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.396097 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.399298 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.403349 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:04.404726 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.420284 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.573297 1039 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:04.575811 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.581701 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.588193 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.598632 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:04.762861 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 977
I20240503 14:12:04.764428 1099 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:04.881930 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 977
I20240503 14:12:05.426442 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.428517 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.434715 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.443048 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.585709 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.586963 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.587251 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.587395 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.752545 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.755354 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.756126 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.758579 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.904503 2092 raft_consensus.cc:482] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b5ad59a974e48c78e85ff29a545dc7e)
I20240503 14:12:05.904924 2092 raft_consensus.cc:504] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } }
I20240503 14:12:05.911477 908 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.914376 2092 leader_election.cc:288] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2:38595), f270bf4de970457dadd6b9f34c70fb05 (127.0.186.4:37377)
I20240503 14:12:05.920598 1429 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.930553 1299 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:05.933880 1169 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240503 14:12:05.943852 1383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.2:38595: connect: Connection refused (error 111)
I20240503 14:12:05.965728 1318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3969333edf443b28972d1b094beeb16" candidate_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "f270bf4de970457dadd6b9f34c70fb05" is_pre_election: true
W20240503 14:12:05.971133 1383 leader_election.cc:334] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2:38595): Network error: Client connection negotiation failed: client connection to 127.0.186.2:38595: connect: Connection refused (error 111)
I20240503 14:12:05.971510 1383 leader_election.cc:302] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 25c5a6b7b8c44ff49ccdb03221a0dba8; no voters: 1b5ad59a974e48c78e85ff29a545dc7e, f270bf4de970457dadd6b9f34c70fb05
I20240503 14:12:05.971999 2092 raft_consensus.cc:2726] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:12:05.994020 2097 raft_consensus.cc:482] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1b5ad59a974e48c78e85ff29a545dc7e)
I20240503 14:12:05.994640 2097 raft_consensus.cc:504] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } }
I20240503 14:12:05.996446 2097 leader_election.cc:288] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2:38595), 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5:44587)
W20240503 14:12:05.997511 1253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.2:38595: connect: Connection refused (error 111)
I20240503 14:12:06.008006 1449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3969333edf443b28972d1b094beeb16" candidate_uuid: "f270bf4de970457dadd6b9f34c70fb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" is_pre_election: true
W20240503 14:12:06.008335 1253 leader_election.cc:334] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2:38595): Network error: Client connection negotiation failed: client connection to 127.0.186.2:38595: connect: Connection refused (error 111)
I20240503 14:12:06.008458 1449 raft_consensus.cc:2454] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f270bf4de970457dadd6b9f34c70fb05 in term 1.
I20240503 14:12:06.009014 1252 leader_election.cc:302] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25c5a6b7b8c44ff49ccdb03221a0dba8, f270bf4de970457dadd6b9f34c70fb05; no voters: 1b5ad59a974e48c78e85ff29a545dc7e
I20240503 14:12:06.009438 2097 raft_consensus.cc:2781] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:12:06.009567 2097 raft_consensus.cc:482] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1b5ad59a974e48c78e85ff29a545dc7e)
I20240503 14:12:06.009652 2097 raft_consensus.cc:3037] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:06.013079 2097 raft_consensus.cc:504] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } }
I20240503 14:12:06.014066 2097 leader_election.cc:288] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [CANDIDATE]: Term 2 election: Requested vote from peers 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2:38595), 25c5a6b7b8c44ff49ccdb03221a0dba8 (127.0.186.5:44587)
I20240503 14:12:06.014672 1449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3969333edf443b28972d1b094beeb16" candidate_uuid: "f270bf4de970457dadd6b9f34c70fb05" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8"
I20240503 14:12:06.014968 1449 raft_consensus.cc:3037] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:12:06.015692 1253 leader_election.cc:334] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2:38595): Network error: Client connection negotiation failed: client connection to 127.0.186.2:38595: connect: Connection refused (error 111)
I20240503 14:12:06.017251 1449 raft_consensus.cc:2454] T b3969333edf443b28972d1b094beeb16 P 25c5a6b7b8c44ff49ccdb03221a0dba8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f270bf4de970457dadd6b9f34c70fb05 in term 2.
I20240503 14:12:06.017735 1252 leader_election.cc:302] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 25c5a6b7b8c44ff49ccdb03221a0dba8, f270bf4de970457dadd6b9f34c70fb05; no voters: 1b5ad59a974e48c78e85ff29a545dc7e
I20240503 14:12:06.018141 2097 raft_consensus.cc:2781] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:06.018728 2097 raft_consensus.cc:686] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [term 2 LEADER]: Becoming Leader. State: Replica: f270bf4de970457dadd6b9f34c70fb05, State: Running, Role: LEADER
I20240503 14:12:06.019286 2097 consensus_queue.cc:237] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } }
I20240503 14:12:06.023192 787 catalog_manager.cc:5484] T b3969333edf443b28972d1b094beeb16 P f270bf4de970457dadd6b9f34c70fb05 reported cstate change: term changed from 1 to 2, leader changed from 1b5ad59a974e48c78e85ff29a545dc7e (127.0.186.2) to f270bf4de970457dadd6b9f34c70fb05 (127.0.186.4). New cstate: current_term: 2 leader_uuid: "f270bf4de970457dadd6b9f34c70fb05" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1b5ad59a974e48c78e85ff29a545dc7e" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 38595 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f270bf4de970457dadd6b9f34c70fb05" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 37377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25c5a6b7b8c44ff49ccdb03221a0dba8" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 44587 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:06.104010 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 846
I20240503 14:12:06.105451 967 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.221643 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 846
I20240503 14:12:06.242746 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 1107
I20240503 14:12:06.243906 1229 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.353093 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 1107
I20240503 14:12:06.372747 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 1237
I20240503 14:12:06.373809 1358 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.492898 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 1237
I20240503 14:12:06.514240 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 1367
I20240503 14:12:06.515393 1489 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.635128 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 1367
I20240503 14:12:06.655274 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 756
I20240503 14:12:06.656520 817 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:06.793259 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 756
2024-05-03T14:12:06Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (11892 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (11892 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240503 14:12:06.841825 744 test_util.cc:268] Using random seed: 1308750254
I20240503 14:12:06.843730 744 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:06.843819 744 ts_itest-base.cc:113] --------------
I20240503 14:12:06.843856 744 ts_itest-base.cc:114] 5 tablet servers
I20240503 14:12:06.843907 744 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:12:06.843945 744 ts_itest-base.cc:116] --------------
2024-05-03T14:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:06Z Disabled control of system clock
I20240503 14:12:06.869169 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.186.62:36949
--webserver_interface=127.0.186.62
--webserver_port=0
--builtin_ntp_servers=127.0.186.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.186.62:36949
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:12:07.208426 2120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:07.208768 2120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:07.208863 2120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:07.216615 2120 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:07.216759 2120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:07.216802 2120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:07.216863 2120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:07.216910 2120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:07.226289 2120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.186.62:36949
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.186.62:36949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.186.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:07.227677 2120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:07.229398 2120 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.241515 2125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.242225 2126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.243084 2128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.244237 2120 server_base.cc:1034] running on GCE node
I20240503 14:12:07.245190 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:07.246899 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:07.248138 2120 hybrid_clock.cc:648] HybridClock initialized: now 1714745527248092 us; error 52 us; skew 500 ppm
I20240503 14:12:07.248533 2120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.251255 2120 webserver.cc:460] Webserver started at http://127.0.186.62:37957/ using document root <none> and password file <none>
I20240503 14:12:07.252122 2120 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.252250 2120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.252604 2120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.257243 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2fecf9a8c3b44a9a835cc4c364b89385"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-mh2c"
I20240503 14:12:07.258129 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2fecf9a8c3b44a9a835cc4c364b89385"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-mh2c"
I20240503 14:12:07.264369 2120 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 14:12:07.268326 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.270030 2120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:07.270252 2120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2fecf9a8c3b44a9a835cc4c364b89385"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-mh2c"
I20240503 14:12:07.270484 2120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.294113 2120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.295033 2120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:07.295316 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.314510 2120 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.62:36949
I20240503 14:12:07.314532 2185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.62:36949 every 8 connection(s)
I20240503 14:12:07.316087 2120 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:07.318488 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 2120
I20240503 14:12:07.318742 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:07.321049 2186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:07.333051 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385: Bootstrap starting.
I20240503 14:12:07.336395 2186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:07.337487 2186 log.cc:824] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:07.340494 2186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385: No bootstrap required, opened a new log
I20240503 14:12:07.346853 2186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fecf9a8c3b44a9a835cc4c364b89385" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 36949 } }
I20240503 14:12:07.347282 2186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:07.347388 2186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fecf9a8c3b44a9a835cc4c364b89385, State: Initialized, Role: FOLLOWER
I20240503 14:12:07.348088 2186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fecf9a8c3b44a9a835cc4c364b89385" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 36949 } }
I20240503 14:12:07.348315 2186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:07.348454 2186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:07.348615 2186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:07.350741 2186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fecf9a8c3b44a9a835cc4c364b89385" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 36949 } }
I20240503 14:12:07.351263 2186 leader_election.cc:302] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fecf9a8c3b44a9a835cc4c364b89385; no voters:
I20240503 14:12:07.351922 2186 leader_election.cc:288] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:07.352070 2191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:07.352808 2191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [term 1 LEADER]: Becoming Leader. State: Replica: 2fecf9a8c3b44a9a835cc4c364b89385, State: Running, Role: LEADER
I20240503 14:12:07.353418 2191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fecf9a8c3b44a9a835cc4c364b89385" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 36949 } }
I20240503 14:12:07.354120 2186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:07.356331 2192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fecf9a8c3b44a9a835cc4c364b89385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fecf9a8c3b44a9a835cc4c364b89385" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 36949 } } }
I20240503 14:12:07.356701 2192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:07.356562 2193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fecf9a8c3b44a9a835cc4c364b89385. Latest consensus state: current_term: 1 leader_uuid: "2fecf9a8c3b44a9a835cc4c364b89385" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fecf9a8c3b44a9a835cc4c364b89385" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 36949 } } }
I20240503 14:12:07.356921 2193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:07.360719 2196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:07.366652 2196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:07.375885 2196 catalog_manager.cc:1347] Generated new cluster ID: fa82e548fdd14e5285ac38924c566b6b
I20240503 14:12:07.376032 2196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:07.396984 2196 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:07.398253 2196 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:07.408710 2196 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2fecf9a8c3b44a9a835cc4c364b89385: Generated new TSK 0
I20240503 14:12:07.409588 2196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:07.421876 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.1:0
--local_ip_for_outbound_sockets=127.0.186.1
--webserver_interface=127.0.186.1
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--builtin_ntp_servers=127.0.186.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:07.783046 2210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:07.783365 2210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:07.783500 2210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:07.791085 2210 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:07.791221 2210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:07.791386 2210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.1
I20240503 14:12:07.800873 2210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.186.1
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.1
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:07.802302 2210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:07.804148 2210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:07.817276 2218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.818094 2215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:07.818897 2216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:07.820034 2210 server_base.cc:1034] running on GCE node
I20240503 14:12:07.820663 2210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:07.821750 2210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:07.822962 2210 hybrid_clock.cc:648] HybridClock initialized: now 1714745527822908 us; error 80 us; skew 500 ppm
I20240503 14:12:07.823348 2210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:07.825871 2210 webserver.cc:460] Webserver started at http://127.0.186.1:36531/ using document root <none> and password file <none>
I20240503 14:12:07.826684 2210 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:07.826808 2210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:07.827152 2210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:07.830010 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bca818be3cbd49429bff69cb08333d23"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-mh2c"
I20240503 14:12:07.830848 2210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bca818be3cbd49429bff69cb08333d23"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-mh2c"
I20240503 14:12:07.836571 2210 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240503 14:12:07.841027 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.842839 2210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240503 14:12:07.843065 2210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bca818be3cbd49429bff69cb08333d23"
format_stamp: "Formatted at 2024-05-03 14:12:07 on dist-test-slave-mh2c"
I20240503 14:12:07.843310 2210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:07.880961 2210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:07.881879 2210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:07.882171 2210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:07.883201 2210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:07.884838 2210 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:07.884963 2210 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.885097 2210 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:07.885174 2210 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:07.920585 2210 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.1:37841
I20240503 14:12:07.920724 2336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.1:37841 every 8 connection(s)
I20240503 14:12:07.923382 2210 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:07.924118 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 2210
I20240503 14:12:07.924353 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:07.928416 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.2:0
--local_ip_for_outbound_sockets=127.0.186.2
--webserver_interface=127.0.186.2
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--builtin_ntp_servers=127.0.186.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:07.938649 2337 heartbeater.cc:347] Connected to a master server at 127.0.186.62:36949
I20240503 14:12:07.939024 2337 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:07.939951 2337 heartbeater.cc:511] Master 127.0.186.62:36949 requested a full tablet report, sending...
I20240503 14:12:07.942319 2151 ts_manager.cc:194] Registered new tserver with Master: bca818be3cbd49429bff69cb08333d23 (127.0.186.1:37841)
I20240503 14:12:07.944007 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.1:37209
W20240503 14:12:08.276453 2341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:08.276803 2341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:08.276952 2341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:08.284674 2341 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:08.284848 2341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:08.285022 2341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.2
I20240503 14:12:08.294303 2341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.186.2
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.2
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:08.295776 2341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:08.297597 2341 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:08.311164 2349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.311910 2346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.312840 2347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:08.313917 2341 server_base.cc:1034] running on GCE node
I20240503 14:12:08.314749 2341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:08.316107 2341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:08.317374 2341 hybrid_clock.cc:648] HybridClock initialized: now 1714745528317306 us; error 67 us; skew 500 ppm
I20240503 14:12:08.317783 2341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:08.320914 2341 webserver.cc:460] Webserver started at http://127.0.186.2:40653/ using document root <none> and password file <none>
I20240503 14:12:08.321985 2341 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:08.322136 2341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:08.322494 2341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:08.326238 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "70f91faefdc74269b86f1b138ab38e8c"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-mh2c"
I20240503 14:12:08.327283 2341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "70f91faefdc74269b86f1b138ab38e8c"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-mh2c"
I20240503 14:12:08.334621 2341 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20240503 14:12:08.338771 2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.340478 2341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:08.340644 2341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "70f91faefdc74269b86f1b138ab38e8c"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-mh2c"
I20240503 14:12:08.340857 2341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.365855 2341 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.366762 2341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:08.367049 2341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.368181 2341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.369742 2341 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.369833 2341 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.369944 2341 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.370013 2341 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.402963 2341 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.2:40019
I20240503 14:12:08.403023 2467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.2:40019 every 8 connection(s)
I20240503 14:12:08.404348 2341 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:08.409590 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 2341
I20240503 14:12:08.409819 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:08.413285 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.3:0
--local_ip_for_outbound_sockets=127.0.186.3
--webserver_interface=127.0.186.3
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--builtin_ntp_servers=127.0.186.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:08.418502 2468 heartbeater.cc:347] Connected to a master server at 127.0.186.62:36949
I20240503 14:12:08.418824 2468 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.419713 2468 heartbeater.cc:511] Master 127.0.186.62:36949 requested a full tablet report, sending...
I20240503 14:12:08.421303 2151 ts_manager.cc:194] Registered new tserver with Master: 70f91faefdc74269b86f1b138ab38e8c (127.0.186.2:40019)
I20240503 14:12:08.422076 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.2:55781
W20240503 14:12:08.758867 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:08.759222 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:08.759378 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:08.767239 2471 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:08.767413 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:08.767623 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.3
I20240503 14:12:08.777137 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.186.3
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.3
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:08.778573 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:08.780740 2471 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:08.794520 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.795682 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:08.796561 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:08.797506 2471 server_base.cc:1034] running on GCE node
I20240503 14:12:08.798166 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:08.799190 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:08.800400 2471 hybrid_clock.cc:648] HybridClock initialized: now 1714745528800351 us; error 42 us; skew 500 ppm
I20240503 14:12:08.800812 2471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:08.803179 2471 webserver.cc:460] Webserver started at http://127.0.186.3:42197/ using document root <none> and password file <none>
I20240503 14:12:08.804031 2471 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:08.804138 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:08.804433 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:08.807173 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1d41324406714840926b361b3d8aa8be"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-mh2c"
I20240503 14:12:08.807996 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1d41324406714840926b361b3d8aa8be"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-mh2c"
I20240503 14:12:08.813436 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:08.817071 2485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.818516 2471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:08.818691 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1d41324406714840926b361b3d8aa8be"
format_stamp: "Formatted at 2024-05-03 14:12:08 on dist-test-slave-mh2c"
I20240503 14:12:08.818907 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:08.832181 2471 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:08.833050 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:08.833345 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:08.834312 2471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:08.835889 2471 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:08.835994 2471 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.836113 2471 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:08.836189 2471 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:08.868458 2471 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.3:43657
I20240503 14:12:08.868505 2597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.3:43657 every 8 connection(s)
I20240503 14:12:08.869812 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:08.872828 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 2471
I20240503 14:12:08.873018 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:08.876832 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.4:0
--local_ip_for_outbound_sockets=127.0.186.4
--webserver_interface=127.0.186.4
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--builtin_ntp_servers=127.0.186.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:08.884847 2598 heartbeater.cc:347] Connected to a master server at 127.0.186.62:36949
I20240503 14:12:08.885178 2598 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:08.886058 2598 heartbeater.cc:511] Master 127.0.186.62:36949 requested a full tablet report, sending...
I20240503 14:12:08.887753 2151 ts_manager.cc:194] Registered new tserver with Master: 1d41324406714840926b361b3d8aa8be (127.0.186.3:43657)
I20240503 14:12:08.889014 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.3:43163
I20240503 14:12:08.947103 2337 heartbeater.cc:503] Master 127.0.186.62:36949 was elected leader, sending a full tablet report...
W20240503 14:12:09.219267 2601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:09.219611 2601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:09.219748 2601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:09.227627 2601 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:09.227927 2601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:09.228082 2601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.4
I20240503 14:12:09.237723 2601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.186.4
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.4
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:09.239050 2601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:09.240933 2601 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:09.254663 2606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.255657 2609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.256445 2607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:09.257282 2601 server_base.cc:1034] running on GCE node
I20240503 14:12:09.257920 2601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:09.258977 2601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:09.260210 2601 hybrid_clock.cc:648] HybridClock initialized: now 1714745529260161 us; error 41 us; skew 500 ppm
I20240503 14:12:09.260565 2601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:09.263115 2601 webserver.cc:460] Webserver started at http://127.0.186.4:41149/ using document root <none> and password file <none>
I20240503 14:12:09.263948 2601 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:09.264055 2601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:09.264343 2601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:09.267072 2601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "031336003f1f421aac133198ef36a04a"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-mh2c"
I20240503 14:12:09.267935 2601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "031336003f1f421aac133198ef36a04a"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-mh2c"
I20240503 14:12:09.273427 2601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:09.277215 2615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.278688 2601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:09.278898 2601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "031336003f1f421aac133198ef36a04a"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-mh2c"
I20240503 14:12:09.279109 2601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:09.304435 2601 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:09.305347 2601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:09.305622 2601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:09.306691 2601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:09.308300 2601 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:09.308401 2601 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.308526 2601 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:09.308599 2601 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.340402 2601 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.4:40225
I20240503 14:12:09.340459 2727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.4:40225 every 8 connection(s)
I20240503 14:12:09.341827 2601 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:09.347098 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 2601
I20240503 14:12:09.347290 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:09.350311 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.5:0
--local_ip_for_outbound_sockets=127.0.186.5
--webserver_interface=127.0.186.5
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--builtin_ntp_servers=127.0.186.20:45775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:09.355244 2728 heartbeater.cc:347] Connected to a master server at 127.0.186.62:36949
I20240503 14:12:09.355566 2728 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:09.356462 2728 heartbeater.cc:511] Master 127.0.186.62:36949 requested a full tablet report, sending...
I20240503 14:12:09.357998 2151 ts_manager.cc:194] Registered new tserver with Master: 031336003f1f421aac133198ef36a04a (127.0.186.4:40225)
I20240503 14:12:09.359181 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.4:39827
I20240503 14:12:09.424665 2468 heartbeater.cc:503] Master 127.0.186.62:36949 was elected leader, sending a full tablet report...
W20240503 14:12:09.697736 2731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:09.698099 2731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:09.698252 2731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:09.705754 2731 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:09.705900 2731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:09.706068 2731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.5
I20240503 14:12:09.715523 2731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:45775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.186.5
--webserver_port=0
--tserver_master_addrs=127.0.186.62:36949
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.5
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:09.716984 2731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:09.718845 2731 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:09.732306 2736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.732990 2737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:09.733991 2739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:09.735029 2731 server_base.cc:1034] running on GCE node
I20240503 14:12:09.735810 2731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:09.736846 2731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:09.738140 2731 hybrid_clock.cc:648] HybridClock initialized: now 1714745529738079 us; error 51 us; skew 500 ppm
I20240503 14:12:09.738487 2731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:09.740897 2731 webserver.cc:460] Webserver started at http://127.0.186.5:45113/ using document root <none> and password file <none>
I20240503 14:12:09.741685 2731 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:09.741811 2731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:09.742161 2731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:09.745007 2731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ba8dded33132489f97b4bf4aa76d340a"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-mh2c"
I20240503 14:12:09.745849 2731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ba8dded33132489f97b4bf4aa76d340a"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-mh2c"
I20240503 14:12:09.751726 2731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:12:09.755668 2745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.757381 2731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:09.757654 2731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ba8dded33132489f97b4bf4aa76d340a"
format_stamp: "Formatted at 2024-05-03 14:12:09 on dist-test-slave-mh2c"
I20240503 14:12:09.757899 2731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:09.784153 2731 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:09.785084 2731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:09.785365 2731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:09.786402 2731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:09.788059 2731 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:09.788151 2731 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.788235 2731 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:09.788290 2731 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:09.820853 2731 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.5:34535
I20240503 14:12:09.820932 2857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.5:34535 every 8 connection(s)
I20240503 14:12:09.822209 2731 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:09.830384 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 2731
I20240503 14:12:09.830528 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:09.835819 2858 heartbeater.cc:347] Connected to a master server at 127.0.186.62:36949
I20240503 14:12:09.836134 2858 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:09.836953 2858 heartbeater.cc:511] Master 127.0.186.62:36949 requested a full tablet report, sending...
I20240503 14:12:09.838343 2151 ts_manager.cc:194] Registered new tserver with Master: ba8dded33132489f97b4bf4aa76d340a (127.0.186.5:34535)
I20240503 14:12:09.839164 2151 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.5:53909
I20240503 14:12:09.846966 744 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:12:09.863999 2151 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57542:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:12:09.891846 2598 heartbeater.cc:503] Master 127.0.186.62:36949 was elected leader, sending a full tablet report...
I20240503 14:12:09.910919 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 5274cb51984d4326a1d88ec11e362944 (DEFAULT_TABLE table=TestTable [id=d42b039212c340eba73cf0312d2d811c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.910990 2793 tablet_service.cc:1467] Processing CreateTablet for tablet 5274cb51984d4326a1d88ec11e362944 (DEFAULT_TABLE table=TestTable [id=d42b039212c340eba73cf0312d2d811c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.912319 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5274cb51984d4326a1d88ec11e362944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.912444 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5274cb51984d4326a1d88ec11e362944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.914839 2272 tablet_service.cc:1467] Processing CreateTablet for tablet 5274cb51984d4326a1d88ec11e362944 (DEFAULT_TABLE table=TestTable [id=d42b039212c340eba73cf0312d2d811c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:09.916486 2272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5274cb51984d4326a1d88ec11e362944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:09.930727 2876 tablet_bootstrap.cc:492] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be: Bootstrap starting.
I20240503 14:12:09.934144 2877 tablet_bootstrap.cc:492] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a: Bootstrap starting.
I20240503 14:12:09.934799 2878 tablet_bootstrap.cc:492] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23: Bootstrap starting.
I20240503 14:12:09.937070 2876 tablet_bootstrap.cc:654] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.938138 2877 tablet_bootstrap.cc:654] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.938474 2876 log.cc:824] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:09.938640 2878 tablet_bootstrap.cc:654] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:09.939348 2877 log.cc:824] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:09.940084 2878 log.cc:824] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:09.942716 2877 tablet_bootstrap.cc:492] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a: No bootstrap required, opened a new log
I20240503 14:12:09.942798 2878 tablet_bootstrap.cc:492] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23: No bootstrap required, opened a new log
I20240503 14:12:09.943051 2878 ts_tablet_manager.cc:1367] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240503 14:12:09.943042 2877 ts_tablet_manager.cc:1367] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:12:09.943526 2876 tablet_bootstrap.cc:492] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be: No bootstrap required, opened a new log
I20240503 14:12:09.943832 2876 ts_tablet_manager.cc:1367] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20240503 14:12:09.951081 2878 raft_consensus.cc:348] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:09.951853 2878 raft_consensus.cc:374] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.952047 2878 raft_consensus.cc:729] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca818be3cbd49429bff69cb08333d23, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.952558 2877 raft_consensus.cc:348] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:09.952904 2876 raft_consensus.cc:348] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:09.953177 2877 raft_consensus.cc:374] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.953070 2878 consensus_queue.cc:260] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:09.953321 2877 raft_consensus.cc:729] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba8dded33132489f97b4bf4aa76d340a, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.953409 2876 raft_consensus.cc:374] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:09.953522 2876 raft_consensus.cc:729] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d41324406714840926b361b3d8aa8be, State: Initialized, Role: FOLLOWER
I20240503 14:12:09.954186 2877 consensus_queue.cc:260] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:09.954185 2876 consensus_queue.cc:260] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:09.955389 2878 ts_tablet_manager.cc:1398] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20240503 14:12:09.956609 2858 heartbeater.cc:503] Master 127.0.186.62:36949 was elected leader, sending a full tablet report...
I20240503 14:12:09.957902 2876 ts_tablet_manager.cc:1398] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20240503 14:12:09.959935 2877 ts_tablet_manager.cc:1398] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20240503 14:12:10.055205 2883 raft_consensus.cc:482] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:10.055522 2883 raft_consensus.cc:504] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:10.057451 2883 leader_election.cc:288] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba8dded33132489f97b4bf4aa76d340a (127.0.186.5:34535), bca818be3cbd49429bff69cb08333d23 (127.0.186.1:37841)
I20240503 14:12:10.065276 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5274cb51984d4326a1d88ec11e362944" candidate_uuid: "1d41324406714840926b361b3d8aa8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca818be3cbd49429bff69cb08333d23" is_pre_election: true
I20240503 14:12:10.065276 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5274cb51984d4326a1d88ec11e362944" candidate_uuid: "1d41324406714840926b361b3d8aa8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8dded33132489f97b4bf4aa76d340a" is_pre_election: true
I20240503 14:12:10.065769 2813 raft_consensus.cc:2454] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d41324406714840926b361b3d8aa8be in term 0.
I20240503 14:12:10.065778 2292 raft_consensus.cc:2454] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d41324406714840926b361b3d8aa8be in term 0.
I20240503 14:12:10.066438 2486 leader_election.cc:302] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d41324406714840926b361b3d8aa8be, ba8dded33132489f97b4bf4aa76d340a; no voters:
I20240503 14:12:10.066841 2883 raft_consensus.cc:2781] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:10.067013 2883 raft_consensus.cc:482] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:10.067127 2883 raft_consensus.cc:3037] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.069247 2883 raft_consensus.cc:504] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:10.069880 2883 leader_election.cc:288] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [CANDIDATE]: Term 1 election: Requested vote from peers ba8dded33132489f97b4bf4aa76d340a (127.0.186.5:34535), bca818be3cbd49429bff69cb08333d23 (127.0.186.1:37841)
I20240503 14:12:10.070410 2813 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5274cb51984d4326a1d88ec11e362944" candidate_uuid: "1d41324406714840926b361b3d8aa8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba8dded33132489f97b4bf4aa76d340a"
I20240503 14:12:10.070479 2292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5274cb51984d4326a1d88ec11e362944" candidate_uuid: "1d41324406714840926b361b3d8aa8be" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca818be3cbd49429bff69cb08333d23"
I20240503 14:12:10.070590 2813 raft_consensus.cc:3037] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.070713 2292 raft_consensus.cc:3037] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:10.072849 2813 raft_consensus.cc:2454] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d41324406714840926b361b3d8aa8be in term 1.
I20240503 14:12:10.073024 2292 raft_consensus.cc:2454] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d41324406714840926b361b3d8aa8be in term 1.
I20240503 14:12:10.073347 2486 leader_election.cc:302] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d41324406714840926b361b3d8aa8be, ba8dded33132489f97b4bf4aa76d340a; no voters:
I20240503 14:12:10.073731 2883 raft_consensus.cc:2781] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 1 FOLLOWER]: Leader election won for term 1
W20240503 14:12:10.074007 2859 tablet.cc:2369] T 5274cb51984d4326a1d88ec11e362944 P ba8dded33132489f97b4bf4aa76d340a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:10.074376 2883 raft_consensus.cc:686] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [term 1 LEADER]: Becoming Leader. State: Replica: 1d41324406714840926b361b3d8aa8be, State: Running, Role: LEADER
I20240503 14:12:10.074945 2883 consensus_queue.cc:237] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } }
I20240503 14:12:10.080060 2149 catalog_manager.cc:5484] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d41324406714840926b361b3d8aa8be (127.0.186.3). New cstate: current_term: 1 leader_uuid: "1d41324406714840926b361b3d8aa8be" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d41324406714840926b361b3d8aa8be" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 43657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bca818be3cbd49429bff69cb08333d23" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 37841 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:10.109136 744 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 14:12:10.111492 744 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bca818be3cbd49429bff69cb08333d23 to finish bootstrapping
I20240503 14:12:10.120715 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 70f91faefdc74269b86f1b138ab38e8c to finish bootstrapping
W20240503 14:12:10.122257 2599 tablet.cc:2369] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:10.130100 744 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1d41324406714840926b361b3d8aa8be to finish bootstrapping
I20240503 14:12:10.137647 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 031336003f1f421aac133198ef36a04a to finish bootstrapping
I20240503 14:12:10.147508 744 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ba8dded33132489f97b4bf4aa76d340a to finish bootstrapping
I20240503 14:12:10.156129 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 2210
I20240503 14:12:10.157233 2332 generic_service.cc:196] Checking for leaks (request via RPC)
W20240503 14:12:10.244879 2338 tablet.cc:2369] T 5274cb51984d4326a1d88ec11e362944 P bca818be3cbd49429bff69cb08333d23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:10.245575 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 2210
I20240503 14:12:10.361833 2728 heartbeater.cc:503] Master 127.0.186.62:36949 was elected leader, sending a full tablet report...
I20240503 14:12:10.462396 2883 consensus_queue.cc:1035] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba8dded33132489f97b4bf4aa76d340a" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:10.471262 2487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.1:37841: connect: Connection refused (error 111)
W20240503 14:12:10.472838 2487 consensus_peers.cc:485] T 5274cb51984d4326a1d88ec11e362944 P 1d41324406714840926b361b3d8aa8be -> Peer bca818be3cbd49429bff69cb08333d23 (127.0.186.1:37841): Couldn't send request to peer bca818be3cbd49429bff69cb08333d23. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.1:37841: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:10.781191 2663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.781436 2793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.781934 2403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.788854 2533 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:10.932205 2663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.935163 2533 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:10.942258 2403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:10.946295 2793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.112107 2663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.114759 2533 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:11.115721 2403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.117059 2793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.743716 2533 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:11.744100 2793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.750257 2663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.756806 2403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.886893 2663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.889101 2533 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:11.899854 2793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:11.901046 2403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:12.069435 2663 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:12.072085 2533 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:12.074676 2793 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:12.080276 2403 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:12.231616 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 2341
I20240503 14:12:12.233146 2462 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.329619 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 2341
I20240503 14:12:12.347777 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 2471
I20240503 14:12:12.348930 2592 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.447355 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 2471
I20240503 14:12:12.466385 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 2601
I20240503 14:12:12.467471 2722 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.560952 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 2601
I20240503 14:12:12.578933 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 2731
I20240503 14:12:12.580089 2852 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.679149 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 2731
I20240503 14:12:12.696763 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 2120
I20240503 14:12:12.697891 2181 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:12.815356 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 2120
2024-05-03T14:12:12Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (6017 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6017 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240503 14:12:12.859158 744 test_util.cc:268] Using random seed: 1314767611
I20240503 14:12:12.861115 744 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:12.861189 744 ts_itest-base.cc:113] --------------
I20240503 14:12:12.861235 744 ts_itest-base.cc:114] 7 tablet servers
I20240503 14:12:12.861292 744 ts_itest-base.cc:115] 1 replicas per TS
I20240503 14:12:12.861356 744 ts_itest-base.cc:116] --------------
2024-05-03T14:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:12Z Disabled control of system clock
I20240503 14:12:12.886582 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.186.62:42623
--webserver_interface=127.0.186.62
--webserver_port=0
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.186.62:42623
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 14:12:13.225708 3100 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 14:12:13.226269 3100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:13.226395 3100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:13.226505 3100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:13.234326 3100 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:13.234479 3100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:13.234588 3100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:13.234658 3100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:13.234746 3100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:13.244205 3100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.186.62:42623
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.186.62:42623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.186.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:13.245599 3100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:13.247429 3100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:13.260094 3106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.260730 3105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.261792 3108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:13.262948 3100 server_base.cc:1034] running on GCE node
I20240503 14:12:13.263924 3100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:13.265489 3100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:13.266747 3100 hybrid_clock.cc:648] HybridClock initialized: now 1714745533266671 us; error 60 us; skew 500 ppm
I20240503 14:12:13.267122 3100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:13.274688 3100 webserver.cc:460] Webserver started at http://127.0.186.62:41059/ using document root <none> and password file <none>
I20240503 14:12:13.275547 3100 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:13.275753 3100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:13.276091 3100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:13.278887 3100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f88c019087b3473da157be5d002064fa"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-mh2c"
I20240503 14:12:13.279693 3100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f88c019087b3473da157be5d002064fa"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-mh2c"
I20240503 14:12:13.285327 3100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:12:13.288949 3114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.290454 3100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240503 14:12:13.290634 3100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f88c019087b3473da157be5d002064fa"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-mh2c"
I20240503 14:12:13.290895 3100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:13.313817 3100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:13.314707 3100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:13.315006 3100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:13.333158 3100 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.62:42623
I20240503 14:12:13.333182 3165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.62:42623 every 8 connection(s)
I20240503 14:12:13.334625 3100 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:13.336441 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 3100
I20240503 14:12:13.336655 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:13.339483 3166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:13.352428 3166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa: Bootstrap starting.
I20240503 14:12:13.355743 3166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:13.356745 3166 log.cc:824] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:13.359485 3166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa: No bootstrap required, opened a new log
I20240503 14:12:13.365737 3166 raft_consensus.cc:348] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c019087b3473da157be5d002064fa" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 42623 } }
I20240503 14:12:13.366113 3166 raft_consensus.cc:374] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:13.366218 3166 raft_consensus.cc:729] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f88c019087b3473da157be5d002064fa, State: Initialized, Role: FOLLOWER
I20240503 14:12:13.366842 3166 consensus_queue.cc:260] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c019087b3473da157be5d002064fa" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 42623 } }
I20240503 14:12:13.367043 3166 raft_consensus.cc:388] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:13.367169 3166 raft_consensus.cc:482] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:13.367326 3166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:13.369428 3166 raft_consensus.cc:504] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c019087b3473da157be5d002064fa" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 42623 } }
I20240503 14:12:13.369925 3166 leader_election.cc:302] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f88c019087b3473da157be5d002064fa; no voters:
I20240503 14:12:13.370569 3166 leader_election.cc:288] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:13.370744 3171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:13.371942 3171 raft_consensus.cc:686] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [term 1 LEADER]: Becoming Leader. State: Replica: f88c019087b3473da157be5d002064fa, State: Running, Role: LEADER
I20240503 14:12:13.372428 3171 consensus_queue.cc:237] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c019087b3473da157be5d002064fa" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 42623 } }
I20240503 14:12:13.372722 3166 sys_catalog.cc:564] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:13.375696 3172 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f88c019087b3473da157be5d002064fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c019087b3473da157be5d002064fa" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 42623 } } }
I20240503 14:12:13.375962 3172 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:13.376611 3173 sys_catalog.cc:455] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader f88c019087b3473da157be5d002064fa. Latest consensus state: current_term: 1 leader_uuid: "f88c019087b3473da157be5d002064fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f88c019087b3473da157be5d002064fa" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 42623 } } }
I20240503 14:12:13.376955 3173 sys_catalog.cc:458] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:13.377656 3179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:13.382092 3179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:13.391088 3179 catalog_manager.cc:1347] Generated new cluster ID: b7f825b758544684a347f11c20c3b40b
I20240503 14:12:13.391218 3179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:13.418406 3179 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:13.419737 3179 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:13.433126 3179 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f88c019087b3473da157be5d002064fa: Generated new TSK 0
I20240503 14:12:13.434087 3179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:13.443348 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.1:0
--local_ip_for_outbound_sockets=127.0.186.1
--webserver_interface=127.0.186.1
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:13.788600 3190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:13.788921 3190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:13.789060 3190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:13.797150 3190 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:13.797297 3190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:13.797467 3190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.1
I20240503 14:12:13.806900 3190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.186.1
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.1
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:13.808368 3190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:13.810184 3190 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:13.823781 3198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.824457 3196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:13.825253 3195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:13.826401 3190 server_base.cc:1034] running on GCE node
I20240503 14:12:13.827062 3190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:13.828197 3190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:13.829423 3190 hybrid_clock.cc:648] HybridClock initialized: now 1714745533829372 us; error 47 us; skew 500 ppm
I20240503 14:12:13.829741 3190 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:13.832406 3190 webserver.cc:460] Webserver started at http://127.0.186.1:33967/ using document root <none> and password file <none>
I20240503 14:12:13.833194 3190 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:13.833302 3190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:13.833599 3190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:13.836449 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f7f1ddc70d7f442296eb038048b1069d"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-mh2c"
I20240503 14:12:13.837234 3190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f7f1ddc70d7f442296eb038048b1069d"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-mh2c"
I20240503 14:12:13.842759 3190 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:13.846693 3204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.848368 3190 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:13.848559 3190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f7f1ddc70d7f442296eb038048b1069d"
format_stamp: "Formatted at 2024-05-03 14:12:13 on dist-test-slave-mh2c"
I20240503 14:12:13.848798 3190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:13.868443 3190 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:13.869370 3190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:13.869658 3190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:13.870631 3190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:13.872215 3190 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:13.872308 3190 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.872426 3190 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:13.872516 3190 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:13.904168 3190 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.1:46751
I20240503 14:12:13.904253 3316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.1:46751 every 8 connection(s)
I20240503 14:12:13.905531 3190 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:13.914271 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 3190
I20240503 14:12:13.914527 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:13.917886 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.2:0
--local_ip_for_outbound_sockets=127.0.186.2
--webserver_interface=127.0.186.2
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:13.919945 3317 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:13.920338 3317 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:13.921284 3317 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:13.923836 3131 ts_manager.cc:194] Registered new tserver with Master: f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:13.925387 3131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.1:33703
W20240503 14:12:14.256620 3321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:14.256953 3321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:14.257102 3321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:14.264819 3321 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:14.264992 3321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:14.265165 3321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.2
I20240503 14:12:14.274400 3321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.186.2
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.2
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:14.275918 3321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:14.277714 3321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.291002 3326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.291579 3327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.292569 3329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.293679 3321 server_base.cc:1034] running on GCE node
I20240503 14:12:14.294392 3321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:14.295512 3321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:14.296710 3321 hybrid_clock.cc:648] HybridClock initialized: now 1714745534296656 us; error 46 us; skew 500 ppm
I20240503 14:12:14.297109 3321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.299504 3321 webserver.cc:460] Webserver started at http://127.0.186.2:38937/ using document root <none> and password file <none>
I20240503 14:12:14.300285 3321 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.300385 3321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.300650 3321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:14.303261 3321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "fb0746b4d18f470f8815143973d6cf4d"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-mh2c"
I20240503 14:12:14.304054 3321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "fb0746b4d18f470f8815143973d6cf4d"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-mh2c"
I20240503 14:12:14.309393 3321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:14.313110 3335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.314716 3321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:14.314913 3321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "fb0746b4d18f470f8815143973d6cf4d"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-mh2c"
I20240503 14:12:14.315138 3321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.346138 3321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.347277 3321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:14.347620 3321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.348773 3321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.350756 3321 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.350872 3321 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.350987 3321 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.351068 3321 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.381662 3321 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.2:41985
I20240503 14:12:14.381724 3447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.2:41985 every 8 connection(s)
I20240503 14:12:14.383034 3321 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:14.388717 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 3321
I20240503 14:12:14.388877 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:14.392055 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.3:0
--local_ip_for_outbound_sockets=127.0.186.3
--webserver_interface=127.0.186.3
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:14.396167 3448 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:14.396582 3448 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.397679 3448 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:14.399194 3131 ts_manager.cc:194] Registered new tserver with Master: fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:14.400007 3131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.2:37095
W20240503 14:12:14.735862 3451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:14.736207 3451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:14.736354 3451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:14.743903 3451 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:14.744068 3451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:14.744227 3451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.3
I20240503 14:12:14.753683 3451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.186.3
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.3
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:14.755132 3451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:14.757009 3451 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:14.770484 3456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.771399 3459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:14.772007 3457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:14.773303 3451 server_base.cc:1034] running on GCE node
I20240503 14:12:14.774051 3451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:14.775188 3451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:14.776460 3451 hybrid_clock.cc:648] HybridClock initialized: now 1714745534776401 us; error 51 us; skew 500 ppm
I20240503 14:12:14.776849 3451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:14.779711 3451 webserver.cc:460] Webserver started at http://127.0.186.3:39125/ using document root <none> and password file <none>
I20240503 14:12:14.780620 3451 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:14.780761 3451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:14.781119 3451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:14.784302 3451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "89194a0dee124ed1a36b1e81aba11e8c"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-mh2c"
I20240503 14:12:14.785245 3451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "89194a0dee124ed1a36b1e81aba11e8c"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-mh2c"
I20240503 14:12:14.791404 3451 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240503 14:12:14.795378 3465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.797209 3451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:14.797403 3451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "89194a0dee124ed1a36b1e81aba11e8c"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-mh2c"
I20240503 14:12:14.797673 3451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:14.825783 3451 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:14.826721 3451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:14.827030 3451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:14.828135 3451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:14.829759 3451 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:14.829865 3451 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.829998 3451 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:14.830073 3451 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:14.863654 3451 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.3:34149
I20240503 14:12:14.863695 3577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.3:34149 every 8 connection(s)
I20240503 14:12:14.864980 3451 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:14.873133 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 3451
I20240503 14:12:14.873344 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:14.876475 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.4:0
--local_ip_for_outbound_sockets=127.0.186.4
--webserver_interface=127.0.186.4
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:14.884522 3578 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:14.884881 3578 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:14.885836 3578 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:14.887415 3131 ts_manager.cc:194] Registered new tserver with Master: 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3:34149)
I20240503 14:12:14.888379 3131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.3:42127
I20240503 14:12:14.928534 3317 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
W20240503 14:12:15.229265 3581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:15.229599 3581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:15.229732 3581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:15.237373 3581 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:15.237509 3581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:15.237672 3581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.4
I20240503 14:12:15.246873 3581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.186.4
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.4
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:15.248350 3581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:15.250206 3581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:15.264201 3587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:15.265234 3586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:15.266093 3589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:15.267092 3581 server_base.cc:1034] running on GCE node
I20240503 14:12:15.268008 3581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:15.269160 3581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:15.270409 3581 hybrid_clock.cc:648] HybridClock initialized: now 1714745535270368 us; error 37 us; skew 500 ppm
I20240503 14:12:15.270778 3581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:15.273342 3581 webserver.cc:460] Webserver started at http://127.0.186.4:42775/ using document root <none> and password file <none>
I20240503 14:12:15.274184 3581 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:15.274307 3581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:15.274649 3581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:15.277477 3581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "ed03e4970b2844df8fb33aa1a1b440c9"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-mh2c"
I20240503 14:12:15.278301 3581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "ed03e4970b2844df8fb33aa1a1b440c9"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-mh2c"
I20240503 14:12:15.283802 3581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 14:12:15.287442 3595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.288851 3581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:15.289038 3581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ed03e4970b2844df8fb33aa1a1b440c9"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-mh2c"
I20240503 14:12:15.289225 3581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:15.322312 3581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:15.323175 3581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:15.323434 3581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:15.324419 3581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:15.325913 3581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:15.325999 3581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.326081 3581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:15.326135 3581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.359438 3581 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.4:39891
I20240503 14:12:15.359530 3707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.4:39891 every 8 connection(s)
I20240503 14:12:15.360944 3581 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:15.367465 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 3581
I20240503 14:12:15.367686 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:15.371027 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.5:0
--local_ip_for_outbound_sockets=127.0.186.5
--webserver_interface=127.0.186.5
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:15.375051 3708 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:15.375387 3708 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:15.376401 3708 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:15.378170 3131 ts_manager.cc:194] Registered new tserver with Master: ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4:39891)
I20240503 14:12:15.379015 3131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.4:34709
I20240503 14:12:15.402586 3448 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
W20240503 14:12:15.719148 3711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:15.719512 3711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:15.719704 3711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:15.727465 3711 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:15.727648 3711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:15.727829 3711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.5
I20240503 14:12:15.737242 3711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.186.5
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.5
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:15.738685 3711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:15.740592 3711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:15.754287 3716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:15.755120 3717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:15.755756 3719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:15.757016 3711 server_base.cc:1034] running on GCE node
I20240503 14:12:15.757705 3711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:15.758792 3711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:15.760100 3711 hybrid_clock.cc:648] HybridClock initialized: now 1714745535760042 us; error 51 us; skew 500 ppm
I20240503 14:12:15.760457 3711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:15.763027 3711 webserver.cc:460] Webserver started at http://127.0.186.5:41153/ using document root <none> and password file <none>
I20240503 14:12:15.763888 3711 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:15.764024 3711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:15.764369 3711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:15.767287 3711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-mh2c"
I20240503 14:12:15.768132 3711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-mh2c"
I20240503 14:12:15.773682 3711 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:12:15.777472 3725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.779021 3711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:15.779227 3711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-mh2c"
I20240503 14:12:15.779443 3711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:15.798172 3711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:15.799078 3711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:15.799376 3711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:15.800428 3711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:15.801964 3711 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:15.802067 3711 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.802179 3711 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:15.802258 3711 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:15.835260 3711 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.5:39315
I20240503 14:12:15.835309 3837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.5:39315 every 8 connection(s)
I20240503 14:12:15.836690 3711 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:15.841814 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 3711
I20240503 14:12:15.842011 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:15.845090 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.6:0
--local_ip_for_outbound_sockets=127.0.186.6
--webserver_interface=127.0.186.6
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:15.850845 3838 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:15.851218 3838 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:15.852254 3838 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:15.853964 3131 ts_manager.cc:194] Registered new tserver with Master: b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5:39315)
I20240503 14:12:15.854765 3131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.5:46853
I20240503 14:12:15.891129 3578 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
W20240503 14:12:16.187115 3841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:16.187453 3841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:16.187551 3841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:16.195297 3841 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:16.195451 3841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:16.195567 3841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.6
I20240503 14:12:16.205152 3841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.186.6
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.6
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:16.206548 3841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:16.208401 3841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:16.221771 3847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:16.222872 3846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:16.223438 3849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:16.224314 3841 server_base.cc:1034] running on GCE node
I20240503 14:12:16.224944 3841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:16.226027 3841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:16.227262 3841 hybrid_clock.cc:648] HybridClock initialized: now 1714745536227204 us; error 51 us; skew 500 ppm
I20240503 14:12:16.227694 3841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:16.230307 3841 webserver.cc:460] Webserver started at http://127.0.186.6:42929/ using document root <none> and password file <none>
I20240503 14:12:16.231127 3841 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:16.231238 3841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:16.231564 3841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:16.234450 3841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6c12721557f44ff28c4be1bcef9244c6"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-mh2c"
I20240503 14:12:16.235266 3841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6c12721557f44ff28c4be1bcef9244c6"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-mh2c"
I20240503 14:12:16.240814 3841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20240503 14:12:16.244632 3855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.246150 3841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:16.246335 3841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c12721557f44ff28c4be1bcef9244c6"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-mh2c"
I20240503 14:12:16.246548 3841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:16.265687 3841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:16.266594 3841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:16.266873 3841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:16.267922 3841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:16.269501 3841 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:16.269600 3841 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.269717 3841 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:16.269789 3841 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.302888 3841 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.6:36477
I20240503 14:12:16.302939 3967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.6:36477 every 8 connection(s)
I20240503 14:12:16.304317 3841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:16.304718 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 3841
I20240503 14:12:16.304875 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:12:16.308707 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.7:0
--local_ip_for_outbound_sockets=127.0.186.7
--webserver_interface=127.0.186.7
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:16.319149 3968 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:16.319511 3968 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:16.320451 3968 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:16.322167 3131 ts_manager.cc:194] Registered new tserver with Master: 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6:36477)
I20240503 14:12:16.323426 3131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.6:46259
I20240503 14:12:16.381697 3708 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
W20240503 14:12:16.664829 3971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:16.665164 3971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:16.665319 3971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:16.673092 3971 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:16.673275 3971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:16.673475 3971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.7
I20240503 14:12:16.682842 3971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.186.7
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.7
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:16.684427 3971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:16.686256 3971 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:16.700006 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:16.700901 3971 server_base.cc:1034] running on GCE node
W20240503 14:12:16.701843 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:16.702864 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:16.703727 3971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:16.704747 3971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:16.706046 3971 hybrid_clock.cc:648] HybridClock initialized: now 1714745536705991 us; error 46 us; skew 500 ppm
I20240503 14:12:16.706413 3971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:16.708956 3971 webserver.cc:460] Webserver started at http://127.0.186.7:41449/ using document root <none> and password file <none>
I20240503 14:12:16.709735 3971 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:16.709851 3971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:16.710178 3971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:16.713043 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-mh2c"
I20240503 14:12:16.713894 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-mh2c"
I20240503 14:12:16.719518 3971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:12:16.723655 3985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.725415 3971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240503 14:12:16.725618 3971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-mh2c"
I20240503 14:12:16.725852 3971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:16.740798 3971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:16.741712 3971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:16.742002 3971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:16.743007 3971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:16.744765 3971 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:16.744856 3971 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.744978 3971 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:16.745051 3971 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:16.777704 3971 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.7:43489
I20240503 14:12:16.777755 4097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.7:43489 every 8 connection(s)
I20240503 14:12:16.779042 3971 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:16.779740 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 3971
I20240503 14:12:16.779981 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:12:16.793040 4098 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:16.793350 4098 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:16.794298 4098 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:16.796111 3130 ts_manager.cc:194] Registered new tserver with Master: f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7:43489)
I20240503 14:12:16.797052 3130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.7:51359
I20240503 14:12:16.809329 744 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:12:16.827484 3130 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60048:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 14:12:16.857472 3838 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
I20240503 14:12:16.856971 3513 tablet_service.cc:1467] Processing CreateTablet for tablet bdf123b6cdb64acfbdb132241ec11de0 (DEFAULT_TABLE table=TestTable [id=0ac6adc511d94bb598fec7b4d6ce2c9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:16.858526 3513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdf123b6cdb64acfbdb132241ec11de0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:16.871381 4112 tablet_bootstrap.cc:492] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Bootstrap starting.
I20240503 14:12:16.874652 4112 tablet_bootstrap.cc:654] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:16.875759 4112 log.cc:824] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:16.878571 4112 tablet_bootstrap.cc:492] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: No bootstrap required, opened a new log
I20240503 14:12:16.878826 4112 ts_tablet_manager.cc:1367] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240503 14:12:16.885017 4112 raft_consensus.cc:348] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } }
I20240503 14:12:16.885547 4112 raft_consensus.cc:374] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:16.885702 4112 raft_consensus.cc:729] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89194a0dee124ed1a36b1e81aba11e8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:16.886534 4112 consensus_queue.cc:260] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } }
I20240503 14:12:16.886785 4112 raft_consensus.cc:388] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:16.886991 4112 raft_consensus.cc:482] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:16.887217 4112 raft_consensus.cc:3037] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:16.889485 4112 raft_consensus.cc:504] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } }
I20240503 14:12:16.890012 4112 leader_election.cc:302] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89194a0dee124ed1a36b1e81aba11e8c; no voters:
I20240503 14:12:16.890630 4112 leader_election.cc:288] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:16.890805 4114 raft_consensus.cc:2781] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:16.891486 4114 raft_consensus.cc:686] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 LEADER]: Becoming Leader. State: Replica: 89194a0dee124ed1a36b1e81aba11e8c, State: Running, Role: LEADER
I20240503 14:12:16.892033 4114 consensus_queue.cc:237] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } }
I20240503 14:12:16.892635 4112 ts_tablet_manager.cc:1398] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20240503 14:12:16.897713 3130 catalog_manager.cc:5484] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "89194a0dee124ed1a36b1e81aba11e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:16.916879 744 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 14:12:16.918992 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f7f1ddc70d7f442296eb038048b1069d to finish bootstrapping
I20240503 14:12:16.929059 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver fb0746b4d18f470f8815143973d6cf4d to finish bootstrapping
I20240503 14:12:16.938715 744 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 89194a0dee124ed1a36b1e81aba11e8c to finish bootstrapping
I20240503 14:12:16.946872 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ed03e4970b2844df8fb33aa1a1b440c9 to finish bootstrapping
I20240503 14:12:16.956436 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b10cbf74e3c34a6c8c1972c44e9d3d40 to finish bootstrapping
I20240503 14:12:16.965623 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6c12721557f44ff28c4be1bcef9244c6 to finish bootstrapping
I20240503 14:12:16.975252 744 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f42dc7cc6eec4c1e9f6b6f7721b5d2d9 to finish bootstrapping
I20240503 14:12:16.985286 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 3451
I20240503 14:12:16.986359 3573 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:17.074267 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 3451
I20240503 14:12:17.092645 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 3581
I20240503 14:12:17.093849 3703 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:17.177182 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 3581
I20240503 14:12:17.193189 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 3711
I20240503 14:12:17.194406 3833 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:17.278069 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 3711
I20240503 14:12:17.294250 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 3841
I20240503 14:12:17.295639 3963 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:17.382781 3968 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
I20240503 14:12:17.387454 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 3841
I20240503 14:12:17.403261 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 3971
I20240503 14:12:17.404428 4093 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:17.487944 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 3971
I20240503 14:12:21.256068 3130 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60048:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:21.276196 3251 tablet_service.cc:1467] Processing CreateTablet for tablet 8acb48457026428da222d183814d769c (DEFAULT_TABLE table=rebalance_test_table_0 [id=f76a7ef0c80846f88671272b90dc8175]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.275759 3383 tablet_service.cc:1467] Processing CreateTablet for tablet 819948cd7188407a8a2210f01a2a4f7d (DEFAULT_TABLE table=rebalance_test_table_0 [id=f76a7ef0c80846f88671272b90dc8175]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.275786 3252 tablet_service.cc:1467] Processing CreateTablet for tablet 5a78ff4930264840b7696ab966699f21 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f76a7ef0c80846f88671272b90dc8175]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:21.277331 3252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a78ff4930264840b7696ab966699f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.277343 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819948cd7188407a8a2210f01a2a4f7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.277779 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8acb48457026428da222d183814d769c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:21.289422 4139 tablet_bootstrap.cc:492] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d: Bootstrap starting.
I20240503 14:12:21.290935 4140 tablet_bootstrap.cc:492] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: Bootstrap starting.
I20240503 14:12:21.293234 4139 tablet_bootstrap.cc:654] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.294520 4140 tablet_bootstrap.cc:654] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.294626 4139 log.cc:824] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:21.295897 4140 log.cc:824] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:21.297849 4139 tablet_bootstrap.cc:492] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d: No bootstrap required, opened a new log
I20240503 14:12:21.298159 4139 ts_tablet_manager.cc:1367] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:12:21.298673 4140 tablet_bootstrap.cc:492] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: No bootstrap required, opened a new log
I20240503 14:12:21.298938 4140 ts_tablet_manager.cc:1367] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240503 14:12:21.308018 4140 raft_consensus.cc:348] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:21.307884 4139 raft_consensus.cc:348] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:21.308568 4140 raft_consensus.cc:374] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.308656 4139 raft_consensus.cc:374] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.308706 4140 raft_consensus.cc:729] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.308765 4139 raft_consensus.cc:729] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.309573 4139 consensus_queue.cc:260] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:21.309573 4140 consensus_queue.cc:260] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:21.309852 4139 raft_consensus.cc:388] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:21.309860 4140 raft_consensus.cc:388] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:21.310005 4139 raft_consensus.cc:482] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:21.310005 4140 raft_consensus.cc:482] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:21.310186 4140 raft_consensus.cc:3037] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.310174 4139 raft_consensus.cc:3037] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.313372 4140 raft_consensus.cc:504] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:21.313375 4139 raft_consensus.cc:504] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:21.314124 4139 leader_election.cc:302] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:21.314131 4140 leader_election.cc:302] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:21.314955 4139 leader_election.cc:288] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:21.315078 4143 raft_consensus.cc:2781] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:21.315536 4140 leader_election.cc:288] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:21.315734 4144 raft_consensus.cc:2781] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:21.323796 4139 ts_tablet_manager.cc:1398] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d: Time spent starting tablet: real 0.025s user 0.014s sys 0.013s
I20240503 14:12:21.323983 4140 ts_tablet_manager.cc:1398] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: Time spent starting tablet: real 0.025s user 0.014s sys 0.012s
I20240503 14:12:21.324424 4140 tablet_bootstrap.cc:492] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d: Bootstrap starting.
I20240503 14:12:21.324509 4143 raft_consensus.cc:686] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Leader. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:21.324826 4144 raft_consensus.cc:686] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Leader. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:21.325237 4143 consensus_queue.cc:237] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:21.325793 4144 consensus_queue.cc:237] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:21.326205 4140 tablet_bootstrap.cc:654] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:21.330952 4140 tablet_bootstrap.cc:492] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d: No bootstrap required, opened a new log
I20240503 14:12:21.331218 4140 ts_tablet_manager.cc:1367] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:12:21.331463 3130 catalog_manager.cc:5484] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f1ddc70d7f442296eb038048b1069d (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:21.332293 4140 raft_consensus.cc:348] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:21.332427 4140 raft_consensus.cc:374] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:21.332489 4140 raft_consensus.cc:729] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Initialized, Role: FOLLOWER
I20240503 14:12:21.332695 4140 consensus_queue.cc:260] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:21.332835 4140 raft_consensus.cc:388] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:21.332883 4140 raft_consensus.cc:482] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:21.332953 4140 raft_consensus.cc:3037] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:21.334053 3131 catalog_manager.cc:5484] T 819948cd7188407a8a2210f01a2a4f7d P fb0746b4d18f470f8815143973d6cf4d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0746b4d18f470f8815143973d6cf4d (127.0.186.2). New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:21.336665 4140 raft_consensus.cc:504] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:21.337016 4140 leader_election.cc:302] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:21.337319 4144 raft_consensus.cc:2781] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:21.337519 4144 raft_consensus.cc:686] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Leader. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:21.337811 4144 consensus_queue.cc:237] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:21.338604 4140 leader_election.cc:288] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:21.339218 4140 ts_tablet_manager.cc:1398] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:12:21.346016 3130 catalog_manager.cc:5484] T 8acb48457026428da222d183814d769c P f7f1ddc70d7f442296eb038048b1069d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f1ddc70d7f442296eb038048b1069d (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:21.695050 4155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:21.695448 4155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:21.703078 4155 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 148.871 ms
time per row: 0.0744354 ms
I20240503 14:12:22.007138 3130 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60048:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:22.016225 3383 tablet_service.cc:1467] Processing CreateTablet for tablet 67f73e79bf0344bf9bbc2b1ecf3196a5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2f3fdd1f0e04c8a8811861948f84daa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.016551 3382 tablet_service.cc:1467] Processing CreateTablet for tablet c09ddc167734479ca650e195092f4eee (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2f3fdd1f0e04c8a8811861948f84daa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.016757 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f73e79bf0344bf9bbc2b1ecf3196a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.016615 3251 tablet_service.cc:1467] Processing CreateTablet for tablet 66432f3f144f40a48c8da0eacf2afe0f (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2f3fdd1f0e04c8a8811861948f84daa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.017056 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66432f3f144f40a48c8da0eacf2afe0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.017074 3382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09ddc167734479ca650e195092f4eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.024652 4203 tablet_bootstrap.cc:492] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d: Bootstrap starting.
I20240503 14:12:22.024816 4202 tablet_bootstrap.cc:492] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: Bootstrap starting.
I20240503 14:12:22.026204 4203 tablet_bootstrap.cc:654] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.026218 4202 tablet_bootstrap.cc:654] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.028275 4203 tablet_bootstrap.cc:492] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d: No bootstrap required, opened a new log
I20240503 14:12:22.028352 4202 tablet_bootstrap.cc:492] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: No bootstrap required, opened a new log
I20240503 14:12:22.028415 4203 ts_tablet_manager.cc:1367] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:22.028491 4202 ts_tablet_manager.cc:1367] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:22.029044 4203 raft_consensus.cc:348] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.029209 4203 raft_consensus.cc:374] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.029310 4203 raft_consensus.cc:729] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.029254 4202 raft_consensus.cc:348] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.029409 4202 raft_consensus.cc:374] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.029508 4202 raft_consensus.cc:729] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.029563 4203 consensus_queue.cc:260] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.029742 4203 raft_consensus.cc:388] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:22.029742 4202 consensus_queue.cc:260] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.029827 4203 raft_consensus.cc:482] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:22.029958 4203 raft_consensus.cc:3037] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.029959 4202 raft_consensus.cc:388] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:22.030028 4202 raft_consensus.cc:482] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:22.030143 4202 raft_consensus.cc:3037] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.031936 4203 raft_consensus.cc:504] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.032177 4203 leader_election.cc:302] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:22.032363 4203 leader_election.cc:288] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:22.032358 4202 raft_consensus.cc:504] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.032444 4189 raft_consensus.cc:2781] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.032923 4203 ts_tablet_manager.cc:1398] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20240503 14:12:22.032642 4202 leader_election.cc:302] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:22.033217 4202 leader_election.cc:288] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:22.033370 4194 raft_consensus.cc:2781] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.033982 4194 raft_consensus.cc:686] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Leader. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:22.034456 4194 consensus_queue.cc:237] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.038333 3130 catalog_manager.cc:5484] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0746b4d18f470f8815143973d6cf4d (127.0.186.2). New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:22.042078 4202 ts_tablet_manager.cc:1398] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
I20240503 14:12:22.042409 4202 tablet_bootstrap.cc:492] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: Bootstrap starting.
I20240503 14:12:22.042330 4189 raft_consensus.cc:686] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Leader. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:22.042709 4189 consensus_queue.cc:237] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.044721 4202 tablet_bootstrap.cc:654] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.047024 4202 tablet_bootstrap.cc:492] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: No bootstrap required, opened a new log
I20240503 14:12:22.047245 4202 ts_tablet_manager.cc:1367] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:22.047633 3128 catalog_manager.cc:5484] T 66432f3f144f40a48c8da0eacf2afe0f P f7f1ddc70d7f442296eb038048b1069d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f1ddc70d7f442296eb038048b1069d (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:22.048089 4202 raft_consensus.cc:348] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.048300 4202 raft_consensus.cc:374] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.048384 4202 raft_consensus.cc:729] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.048629 4202 consensus_queue.cc:260] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.048794 4202 raft_consensus.cc:388] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:22.048895 4202 raft_consensus.cc:482] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:22.048990 4202 raft_consensus.cc:3037] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.051733 4202 raft_consensus.cc:504] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.052058 4202 leader_election.cc:302] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:22.052320 4202 leader_election.cc:288] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:22.052445 4207 raft_consensus.cc:2781] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.052610 4207 raft_consensus.cc:686] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Leader. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:22.052925 4202 ts_tablet_manager.cc:1398] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:12:22.052891 4207 consensus_queue.cc:237] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.055454 3130 catalog_manager.cc:5484] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0746b4d18f470f8815143973d6cf4d (127.0.186.2). New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:22.416812 4213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:22.417196 4213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:22.424842 4213 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 125.368 ms
time per row: 0.0626839 ms
I20240503 14:12:22.695216 3130 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60048:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:22.704500 3251 tablet_service.cc:1467] Processing CreateTablet for tablet 291abe2ff8214a3bac8cb9c98b8b20c2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c7b0ffc97dca41f2bc90a97e1631f1ec]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.704694 3382 tablet_service.cc:1467] Processing CreateTablet for tablet 04bc99e3227e47779eeaa86a65a2f51c (DEFAULT_TABLE table=rebalance_test_table_2 [id=c7b0ffc97dca41f2bc90a97e1631f1ec]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.705073 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291abe2ff8214a3bac8cb9c98b8b20c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.705019 3252 tablet_service.cc:1467] Processing CreateTablet for tablet 9dca255e05f4408e82caa70e2dc72af6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c7b0ffc97dca41f2bc90a97e1631f1ec]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:22.705134 3382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04bc99e3227e47779eeaa86a65a2f51c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.705391 3252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dca255e05f4408e82caa70e2dc72af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:22.711025 4258 tablet_bootstrap.cc:492] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d: Bootstrap starting.
I20240503 14:12:22.712302 4259 tablet_bootstrap.cc:492] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: Bootstrap starting.
I20240503 14:12:22.712960 4258 tablet_bootstrap.cc:654] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.713864 4259 tablet_bootstrap.cc:654] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.715479 4258 tablet_bootstrap.cc:492] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d: No bootstrap required, opened a new log
I20240503 14:12:22.715700 4258 ts_tablet_manager.cc:1367] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:22.716168 4259 tablet_bootstrap.cc:492] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: No bootstrap required, opened a new log
I20240503 14:12:22.716315 4259 ts_tablet_manager.cc:1367] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:22.716552 4258 raft_consensus.cc:348] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.716744 4258 raft_consensus.cc:374] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.716841 4258 raft_consensus.cc:729] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.717094 4258 consensus_queue.cc:260] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.717109 4259 raft_consensus.cc:348] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.717288 4259 raft_consensus.cc:374] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.717316 4258 raft_consensus.cc:388] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:22.717386 4259 raft_consensus.cc:729] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.717417 4258 raft_consensus.cc:482] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:22.717531 4258 raft_consensus.cc:3037] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.717631 4259 consensus_queue.cc:260] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.717836 4259 raft_consensus.cc:388] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:22.717916 4259 raft_consensus.cc:482] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:22.718035 4259 raft_consensus.cc:3037] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.720444 4258 raft_consensus.cc:504] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.720749 4258 leader_election.cc:302] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:22.720750 4259 raft_consensus.cc:504] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.720966 4258 leader_election.cc:288] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:22.720999 4259 leader_election.cc:302] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:22.721077 4247 raft_consensus.cc:2781] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.721431 4258 ts_tablet_manager.cc:1398] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:22.721343 4251 raft_consensus.cc:2781] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.721683 4247 raft_consensus.cc:686] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Leader. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:22.721949 4251 raft_consensus.cc:686] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Leader. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:22.721992 4247 consensus_queue.cc:237] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:22.722280 4251 consensus_queue.cc:237] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.721175 4259 leader_election.cc:288] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:22.724144 4259 ts_tablet_manager.cc:1398] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: Time spent starting tablet: real 0.008s user 0.000s sys 0.005s
I20240503 14:12:22.724423 4259 tablet_bootstrap.cc:492] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: Bootstrap starting.
I20240503 14:12:22.724759 3130 catalog_manager.cc:5484] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f1ddc70d7f442296eb038048b1069d (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:22.725699 4259 tablet_bootstrap.cc:654] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:22.726188 3126 catalog_manager.cc:5484] T 04bc99e3227e47779eeaa86a65a2f51c P fb0746b4d18f470f8815143973d6cf4d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0746b4d18f470f8815143973d6cf4d (127.0.186.2). New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:22.727527 4259 tablet_bootstrap.cc:492] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: No bootstrap required, opened a new log
I20240503 14:12:22.727813 4259 ts_tablet_manager.cc:1367] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240503 14:12:22.728883 4259 raft_consensus.cc:348] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.729055 4259 raft_consensus.cc:374] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:22.729138 4259 raft_consensus.cc:729] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Initialized, Role: FOLLOWER
I20240503 14:12:22.729375 4259 consensus_queue.cc:260] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.729514 4259 raft_consensus.cc:388] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:22.729615 4259 raft_consensus.cc:482] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:22.729708 4259 raft_consensus.cc:3037] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:22.731978 4259 raft_consensus.cc:504] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.732312 4259 leader_election.cc:302] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:22.732509 4259 leader_election.cc:288] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:22.732590 4263 raft_consensus.cc:2781] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:22.733091 4263 raft_consensus.cc:686] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Leader. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:22.733417 4263 consensus_queue.cc:237] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:22.734530 4259 ts_tablet_manager.cc:1398] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:12:22.735435 3130 catalog_manager.cc:5484] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f1ddc70d7f442296eb038048b1069d (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:23.092792 4269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:23.093139 4269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:23.101079 4269 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 149.472 ms
time per row: 0.0747359 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:24.330127 3123 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60048:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:24.340528 3252 tablet_service.cc:1467] Processing CreateTablet for tablet 2ce9f735e2044c41900129f35549e7f4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a7ffaa5ef2a24eabac1144221dcd708c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.340479 3382 tablet_service.cc:1467] Processing CreateTablet for tablet d269db92e59e400db4db334fe177c4e6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a7ffaa5ef2a24eabac1144221dcd708c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.340674 3383 tablet_service.cc:1467] Processing CreateTablet for tablet 8c8e2521b6e840ec880476c043408651 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a7ffaa5ef2a24eabac1144221dcd708c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:24.340972 3252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce9f735e2044c41900129f35549e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.340973 3382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d269db92e59e400db4db334fe177c4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.341178 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c8e2521b6e840ec880476c043408651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:24.354688 4318 tablet_bootstrap.cc:492] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: Bootstrap starting.
I20240503 14:12:24.354715 4317 tablet_bootstrap.cc:492] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: Bootstrap starting.
I20240503 14:12:24.356407 4317 tablet_bootstrap.cc:654] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.356441 4318 tablet_bootstrap.cc:654] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.358076 4317 tablet_bootstrap.cc:492] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: No bootstrap required, opened a new log
I20240503 14:12:24.358098 4318 tablet_bootstrap.cc:492] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: No bootstrap required, opened a new log
I20240503 14:12:24.358214 4318 ts_tablet_manager.cc:1367] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:24.358222 4317 ts_tablet_manager.cc:1367] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:24.358825 4318 raft_consensus.cc:348] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:24.358875 4317 raft_consensus.cc:348] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:24.358970 4318 raft_consensus.cc:374] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.359025 4317 raft_consensus.cc:374] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.359056 4318 raft_consensus.cc:729] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.359127 4317 raft_consensus.cc:729] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.359297 4318 consensus_queue.cc:260] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:24.359349 4317 consensus_queue.cc:260] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:24.359458 4318 raft_consensus.cc:388] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:24.359505 4317 raft_consensus.cc:388] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:24.359541 4318 raft_consensus.cc:482] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:24.359627 4317 raft_consensus.cc:482] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:24.359665 4318 raft_consensus.cc:3037] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.359738 4317 raft_consensus.cc:3037] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.361722 4317 raft_consensus.cc:504] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:24.361861 4318 raft_consensus.cc:504] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:24.361950 4317 leader_election.cc:302] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:24.362104 4318 leader_election.cc:302] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:24.362421 4317 leader_election.cc:288] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:24.362558 4321 raft_consensus.cc:2781] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:24.362567 4318 leader_election.cc:288] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:24.362701 4322 raft_consensus.cc:2781] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:24.368202 4322 raft_consensus.cc:686] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Leader. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:24.368763 4322 consensus_queue.cc:237] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:24.369478 4317 ts_tablet_manager.cc:1398] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20240503 14:12:24.369652 4318 ts_tablet_manager.cc:1398] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: Time spent starting tablet: real 0.011s user 0.001s sys 0.011s
I20240503 14:12:24.370110 4318 tablet_bootstrap.cc:492] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: Bootstrap starting.
I20240503 14:12:24.371105 4321 raft_consensus.cc:686] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Leader. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:24.371498 4321 consensus_queue.cc:237] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:24.372216 4318 tablet_bootstrap.cc:654] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:24.372395 3123 catalog_manager.cc:5484] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0746b4d18f470f8815143973d6cf4d (127.0.186.2). New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:24.377909 4318 tablet_bootstrap.cc:492] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: No bootstrap required, opened a new log
I20240503 14:12:24.378165 4318 ts_tablet_manager.cc:1367] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20240503 14:12:24.379235 4318 raft_consensus.cc:348] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:24.379629 4318 raft_consensus.cc:374] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:24.379745 4318 raft_consensus.cc:729] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Initialized, Role: FOLLOWER
I20240503 14:12:24.380049 4318 consensus_queue.cc:260] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:24.380270 4318 raft_consensus.cc:388] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:24.380403 4318 raft_consensus.cc:482] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:24.380522 4318 raft_consensus.cc:3037] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:24.384380 4318 raft_consensus.cc:504] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:24.384732 4318 leader_election.cc:302] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:24.385052 4318 leader_election.cc:288] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:24.385111 4322 raft_consensus.cc:2781] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:24.385393 4322 raft_consensus.cc:686] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Leader. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:24.385932 4322 consensus_queue.cc:237] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:24.386157 4318 ts_tablet_manager.cc:1398] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20240503 14:12:24.389662 3125 catalog_manager.cc:5484] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0746b4d18f470f8815143973d6cf4d (127.0.186.2). New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:24.390000 3123 catalog_manager.cc:5484] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f1ddc70d7f442296eb038048b1069d (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:24.749213 4334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:24.749568 4334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:24.757299 4334 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 147.973 ms
time per row: 0.0739867 ms
I20240503 14:12:25.053417 3128 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60048:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:25.062605 3252 tablet_service.cc:1467] Processing CreateTablet for tablet b6b48c2a49e249348b2a8c5109d251ce (DEFAULT_TABLE table=rebalance_test_table_4 [id=50e14958ddc64d79a9d091ec5d86e98c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.062777 3383 tablet_service.cc:1467] Processing CreateTablet for tablet c67b31603ba3418c99839bf4d9cc6eb0 (DEFAULT_TABLE table=rebalance_test_table_4 [id=50e14958ddc64d79a9d091ec5d86e98c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.063140 3252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b48c2a49e249348b2a8c5109d251ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.063239 3383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67b31603ba3418c99839bf4d9cc6eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.063457 3251 tablet_service.cc:1467] Processing CreateTablet for tablet 551e225f487545e9bfbdde406910e735 (DEFAULT_TABLE table=rebalance_test_table_4 [id=50e14958ddc64d79a9d091ec5d86e98c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:25.063841 3251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551e225f487545e9bfbdde406910e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:25.071844 4381 tablet_bootstrap.cc:492] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: Bootstrap starting.
I20240503 14:12:25.074353 4381 tablet_bootstrap.cc:654] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.076378 4383 tablet_bootstrap.cc:492] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: Bootstrap starting.
I20240503 14:12:25.076705 4381 tablet_bootstrap.cc:492] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: No bootstrap required, opened a new log
I20240503 14:12:25.077055 4381 ts_tablet_manager.cc:1367] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20240503 14:12:25.078042 4381 raft_consensus.cc:348] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:25.078444 4381 raft_consensus.cc:374] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.078619 4381 raft_consensus.cc:729] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.078716 4383 tablet_bootstrap.cc:654] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.078884 4381 consensus_queue.cc:260] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:25.079064 4381 raft_consensus.cc:388] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:25.079144 4381 raft_consensus.cc:482] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:25.079249 4381 raft_consensus.cc:3037] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.081609 4383 tablet_bootstrap.cc:492] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: No bootstrap required, opened a new log
I20240503 14:12:25.081796 4383 ts_tablet_manager.cc:1367] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:25.081892 4381 raft_consensus.cc:504] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:25.082152 4381 leader_election.cc:302] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:25.082346 4381 leader_election.cc:288] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:25.082430 4370 raft_consensus.cc:2781] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.082741 4383 raft_consensus.cc:348] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:25.082943 4383 raft_consensus.cc:374] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.083038 4383 raft_consensus.cc:729] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.083295 4383 consensus_queue.cc:260] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:25.083619 4383 raft_consensus.cc:388] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:25.083729 4383 raft_consensus.cc:482] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:25.083881 4383 raft_consensus.cc:3037] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.083890 4370 raft_consensus.cc:686] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Leader. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:25.084236 4370 consensus_queue.cc:237] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:25.087754 4383 raft_consensus.cc:504] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:25.088197 4383 leader_election.cc:302] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:25.088532 4383 leader_election.cc:288] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:25.088665 4369 raft_consensus.cc:2781] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.088997 4381 ts_tablet_manager.cc:1398] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20240503 14:12:25.089095 4383 ts_tablet_manager.cc:1398] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20240503 14:12:25.089293 4381 tablet_bootstrap.cc:492] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: Bootstrap starting.
I20240503 14:12:25.089797 4369 raft_consensus.cc:686] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Leader. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:25.090157 4369 consensus_queue.cc:237] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } }
I20240503 14:12:25.092141 4381 tablet_bootstrap.cc:654] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:25.094866 4381 tablet_bootstrap.cc:492] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: No bootstrap required, opened a new log
I20240503 14:12:25.095068 4381 ts_tablet_manager.cc:1367] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:12:25.096001 4381 raft_consensus.cc:348] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:25.096289 4381 raft_consensus.cc:374] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:25.096424 4381 raft_consensus.cc:729] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Initialized, Role: FOLLOWER
I20240503 14:12:25.096818 4381 consensus_queue.cc:260] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:25.096783 3128 catalog_manager.cc:5484] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f1ddc70d7f442296eb038048b1069d (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:25.097079 4381 raft_consensus.cc:388] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:25.097175 4381 raft_consensus.cc:482] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:25.097259 4381 raft_consensus.cc:3037] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:25.101166 4381 raft_consensus.cc:504] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:25.101475 4381 leader_election.cc:302] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:25.100879 3124 catalog_manager.cc:5484] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0746b4d18f470f8815143973d6cf4d (127.0.186.2). New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:25.101874 4370 raft_consensus.cc:2781] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:25.102059 4370 raft_consensus.cc:686] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Leader. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:25.102332 4370 consensus_queue.cc:237] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } }
I20240503 14:12:25.103235 4381 leader_election.cc:288] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:25.103874 4381 ts_tablet_manager.cc:1398] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:12:25.106741 3128 catalog_manager.cc:5484] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: term changed from 0 to 1, leader changed from <none> to f7f1ddc70d7f442296eb038048b1069d (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:25.471230 4392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:25.471621 4392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:25.479712 4392 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 148.859 ms
time per row: 0.0744296 ms
I20240503 14:12:25.776557 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.3:34149
--local_ip_for_outbound_sockets=127.0.186.3
--tserver_master_addrs=127.0.186.62:42623
--webserver_port=39125
--webserver_interface=127.0.186.3
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:26.120564 4434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:26.120882 4434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:26.121017 4434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:26.129009 4434 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:26.129160 4434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:26.129339 4434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.3
I20240503 14:12:26.138873 4434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.3:34149
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.186.3
--webserver_port=39125
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.3
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:26.140393 4434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:26.142238 4434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:26.155983 4440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.156926 4442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.157721 4439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:26.158732 4434 server_base.cc:1034] running on GCE node
I20240503 14:12:26.159409 4434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:26.160547 4434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:26.161764 4434 hybrid_clock.cc:648] HybridClock initialized: now 1714745546161697 us; error 59 us; skew 500 ppm
I20240503 14:12:26.162137 4434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:26.164670 4434 webserver.cc:460] Webserver started at http://127.0.186.3:39125/ using document root <none> and password file <none>
I20240503 14:12:26.165455 4434 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:26.165587 4434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:26.171133 4434 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:12:26.174366 4448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.176056 4434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 14:12:26.176247 4434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "89194a0dee124ed1a36b1e81aba11e8c"
format_stamp: "Formatted at 2024-05-03 14:12:14 on dist-test-slave-mh2c"
I20240503 14:12:26.177163 4434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:26.204144 4434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:26.205030 4434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:26.205374 4434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:26.206360 4434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:26.208405 4455 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 14:12:26.213783 4434 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 14:12:26.213900 4434 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240503 14:12:26.214042 4434 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 14:12:26.217149 4434 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 14:12:26.217252 4434 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:26.217619 4455 tablet_bootstrap.cc:492] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Bootstrap starting.
I20240503 14:12:26.237107 4455 log.cc:824] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:26.243391 4455 tablet_bootstrap.cc:492] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:26.244642 4455 tablet_bootstrap.cc:492] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Bootstrap complete.
I20240503 14:12:26.246694 4455 ts_tablet_manager.cc:1367] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.014s
I20240503 14:12:26.255290 4434 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.3:34149
I20240503 14:12:26.255376 4562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.3:34149 every 8 connection(s)
I20240503 14:12:26.257334 4434 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:26.260138 4455 raft_consensus.cc:348] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } }
I20240503 14:12:26.261504 4455 raft_consensus.cc:729] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89194a0dee124ed1a36b1e81aba11e8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:26.262403 4455 consensus_queue.cc:260] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } }
I20240503 14:12:26.262651 4455 raft_consensus.cc:388] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:26.262836 4455 raft_consensus.cc:482] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:26.263049 4455 raft_consensus.cc:3037] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:26.266789 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 4434
I20240503 14:12:26.267575 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.4:39891
--local_ip_for_outbound_sockets=127.0.186.4
--tserver_master_addrs=127.0.186.62:42623
--webserver_port=42775
--webserver_interface=127.0.186.4
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:26.268136 4455 raft_consensus.cc:504] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } }
I20240503 14:12:26.268795 4455 leader_election.cc:302] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89194a0dee124ed1a36b1e81aba11e8c; no voters:
I20240503 14:12:26.269608 4455 leader_election.cc:288] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 14:12:26.269800 4567 raft_consensus.cc:2781] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:26.271689 4567 raft_consensus.cc:686] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [term 2 LEADER]: Becoming Leader. State: Replica: 89194a0dee124ed1a36b1e81aba11e8c, State: Running, Role: LEADER
I20240503 14:12:26.272186 4563 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:26.272542 4563 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:26.273607 4455 ts_tablet_manager.cc:1398] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c: Time spent starting tablet: real 0.027s user 0.019s sys 0.008s
I20240503 14:12:26.273957 4563 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:26.272541 4567 consensus_queue.cc:237] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } }
I20240503 14:12:26.278712 3124 ts_manager.cc:194] Re-registered known tserver with Master: 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3:34149)
I20240503 14:12:26.279138 3124 catalog_manager.cc:5484] T bdf123b6cdb64acfbdb132241ec11de0 P 89194a0dee124ed1a36b1e81aba11e8c reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "89194a0dee124ed1a36b1e81aba11e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:26.284232 3124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.3:46193
I20240503 14:12:26.286919 4563 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
W20240503 14:12:26.618620 4570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:26.618943 4570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:26.619078 4570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:26.627161 4570 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:26.627321 4570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:26.627475 4570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.4
I20240503 14:12:26.636797 4570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.4:39891
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.186.4
--webserver_port=42775
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.4
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:26.638231 4570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:26.640151 4570 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:26.654187 4581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.654979 4582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:26.655649 4584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:26.656773 4570 server_base.cc:1034] running on GCE node
I20240503 14:12:26.657421 4570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:26.658470 4570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:26.659710 4570 hybrid_clock.cc:648] HybridClock initialized: now 1714745546659658 us; error 39 us; skew 500 ppm
I20240503 14:12:26.660041 4570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:26.662483 4570 webserver.cc:460] Webserver started at http://127.0.186.4:42775/ using document root <none> and password file <none>
I20240503 14:12:26.663255 4570 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:26.663398 4570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:26.668968 4570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20240503 14:12:26.671991 4590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.673442 4570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:12:26.673616 4570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "ed03e4970b2844df8fb33aa1a1b440c9"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-mh2c"
I20240503 14:12:26.674525 4570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:26.713304 4570 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:26.714239 4570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:26.714522 4570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:26.715512 4570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:26.717103 4570 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:26.717195 4570 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.717321 4570 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:26.717375 4570 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:26.751475 4570 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.4:39891
I20240503 14:12:26.751571 4702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.4:39891 every 8 connection(s)
I20240503 14:12:26.753033 4570 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:26.762102 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 4570
I20240503 14:12:26.762668 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.5:39315
--local_ip_for_outbound_sockets=127.0.186.5
--tserver_master_addrs=127.0.186.62:42623
--webserver_port=41153
--webserver_interface=127.0.186.5
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:26.773131 4703 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:26.773535 4703 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:26.774683 4703 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:26.776319 3124 ts_manager.cc:194] Re-registered known tserver with Master: ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4:39891)
I20240503 14:12:26.777236 3124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.4:54913
W20240503 14:12:27.109719 4706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:27.110060 4706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:27.110216 4706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:27.118732 4706 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:27.118923 4706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:27.119103 4706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.5
I20240503 14:12:27.128775 4706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.5:39315
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.186.5
--webserver_port=41153
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.5
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:27.130173 4706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:27.131969 4706 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:27.146010 4712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:27.146751 4711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:27.147521 4714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:27.148763 4706 server_base.cc:1034] running on GCE node
I20240503 14:12:27.149526 4706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:27.150578 4706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:27.151803 4706 hybrid_clock.cc:648] HybridClock initialized: now 1714745547151729 us; error 57 us; skew 500 ppm
I20240503 14:12:27.152164 4706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:27.154664 4706 webserver.cc:460] Webserver started at http://127.0.186.5:41153/ using document root <none> and password file <none>
I20240503 14:12:27.155479 4706 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:27.155652 4706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:27.161118 4706 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20240503 14:12:27.164003 4720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.165380 4706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 14:12:27.165568 4706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40"
format_stamp: "Formatted at 2024-05-03 14:12:15 on dist-test-slave-mh2c"
I20240503 14:12:27.166442 4706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:27.185472 4706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:27.186319 4706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:27.186616 4706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:27.187629 4706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:27.189283 4706 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:27.189356 4706 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.189486 4706 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:27.189538 4706 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.222319 4706 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.5:39315
I20240503 14:12:27.222357 4832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.5:39315 every 8 connection(s)
I20240503 14:12:27.223879 4706 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:27.233604 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 4706
I20240503 14:12:27.234323 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.6:36477
--local_ip_for_outbound_sockets=127.0.186.6
--tserver_master_addrs=127.0.186.62:42623
--webserver_port=42929
--webserver_interface=127.0.186.6
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:27.236822 4833 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:27.237141 4833 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:27.238018 4833 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:27.239722 3124 ts_manager.cc:194] Re-registered known tserver with Master: b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5:39315)
I20240503 14:12:27.240494 3124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.5:43493
W20240503 14:12:27.584067 4836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:27.584419 4836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:27.584580 4836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:27.592247 4836 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:27.592401 4836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:27.592581 4836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.6
I20240503 14:12:27.602074 4836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.6:36477
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.186.6
--webserver_port=42929
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.6
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:27.603535 4836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:27.605525 4836 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:27.619053 4842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:27.619851 4841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:27.620744 4844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:27.621760 4836 server_base.cc:1034] running on GCE node
I20240503 14:12:27.622396 4836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:27.623423 4836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:27.624657 4836 hybrid_clock.cc:648] HybridClock initialized: now 1714745547624607 us; error 66 us; skew 500 ppm
I20240503 14:12:27.625033 4836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:27.627532 4836 webserver.cc:460] Webserver started at http://127.0.186.6:42929/ using document root <none> and password file <none>
I20240503 14:12:27.628371 4836 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:27.628507 4836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:27.634301 4836 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20240503 14:12:27.637378 4850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.638924 4836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:27.639106 4836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6c12721557f44ff28c4be1bcef9244c6"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-mh2c"
I20240503 14:12:27.640056 4836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:27.653611 4836 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:27.654510 4836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:27.654793 4836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:27.655813 4836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:27.657389 4836 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:27.657471 4836 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.657606 4836 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:27.657665 4836 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:27.689327 4836 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.6:36477
I20240503 14:12:27.689363 4962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.6:36477 every 8 connection(s)
I20240503 14:12:27.690884 4836 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:27.694636 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 4836
I20240503 14:12:27.695349 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.7:43489
--local_ip_for_outbound_sockets=127.0.186.7
--tserver_master_addrs=127.0.186.62:42623
--webserver_port=41449
--webserver_interface=127.0.186.7
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:27.704703 4963 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:27.705044 4963 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:27.705907 4963 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:27.707656 3124 ts_manager.cc:194] Re-registered known tserver with Master: 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6:36477)
I20240503 14:12:27.708925 3124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.6:38853
I20240503 14:12:27.780161 4703 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
W20240503 14:12:28.046633 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:28.046967 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:28.047112 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:28.055119 4966 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:28.055285 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:28.055466 4966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.7
I20240503 14:12:28.065049 4966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.7:43489
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.186.7
--webserver_port=41449
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.7
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:28.066459 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:28.068339 4966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:28.082594 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:28.083400 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:28.084333 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:28.085399 4966 server_base.cc:1034] running on GCE node
I20240503 14:12:28.086061 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:28.087074 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:28.088317 4966 hybrid_clock.cc:648] HybridClock initialized: now 1714745548088261 us; error 44 us; skew 500 ppm
I20240503 14:12:28.088721 4966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:28.091411 4966 webserver.cc:460] Webserver started at http://127.0.186.7:41449/ using document root <none> and password file <none>
I20240503 14:12:28.092289 4966 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:28.092414 4966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:28.098428 4966 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20240503 14:12:28.101724 4980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:28.103397 4966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:28.103631 4966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9"
format_stamp: "Formatted at 2024-05-03 14:12:16 on dist-test-slave-mh2c"
I20240503 14:12:28.104874 4966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:28.126053 4966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:28.126916 4966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:28.127187 4966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:28.128206 4966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:28.129869 4966 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:28.129948 4966 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:28.130044 4966 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:28.130087 4966 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:28.162549 4966 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.7:43489
I20240503 14:12:28.162565 5092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.7:43489 every 8 connection(s)
I20240503 14:12:28.164104 4966 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:28.165666 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 4966
I20240503 14:12:28.179034 5093 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:28.179366 5093 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:28.180611 5093 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:28.182343 3124 ts_manager.cc:194] Re-registered known tserver with Master: f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7:43489)
I20240503 14:12:28.183641 3124 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.7:58585
I20240503 14:12:28.243754 4833 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
I20240503 14:12:28.698089 4638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.698405 4898 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.711730 4963 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
I20240503 14:12:28.719771 4497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:28.724593 3383 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:28.726748 4768 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.729179 5028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.732041 3252 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:12:28.954530 4898 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.964006 4497 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:12:28.975045 4638 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:28.977715 4768 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.007699 3252 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:12:29.018818 5028 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:29.030165 3383 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:12:29.094818 3403 consensus_queue.cc:237] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } }
I20240503 14:12:29.097720 5184 raft_consensus.cc:2932] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } } }
I20240503 14:12:29.104292 3124 catalog_manager.cc:5484] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index -1 to 4, NON_VOTER b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:29.107625 3336 consensus_peers.cc:485] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d -> Peer b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5:39315): Couldn't send request to peer b10cbf74e3c34a6c8c1972c44e9d3d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d269db92e59e400db4db334fe177c4e6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.117039 3272 consensus_queue.cc:237] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } }
I20240503 14:12:29.119769 5197 raft_consensus.cc:2932] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } } }
I20240503 14:12:29.127023 3124 catalog_manager.cc:5484] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index -1 to 4, NON_VOTER ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:29.128901 3208 consensus_peers.cc:485] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d -> Peer ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4:39891): Couldn't send request to peer ed03e4970b2844df8fb33aa1a1b440c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6b48c2a49e249348b2a8c5109d251ce. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.138561 3403 consensus_queue.cc:237] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } }
I20240503 14:12:29.140304 5183 raft_consensus.cc:2932] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } } }
I20240503 14:12:29.146194 3128 catalog_manager.cc:5484] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index -1 to 4, NON_VOTER f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:29.147781 3337 consensus_peers.cc:485] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d -> Peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7:43489): Couldn't send request to peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c09ddc167734479ca650e195092f4eee. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.157181 3272 consensus_queue.cc:237] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } }
I20240503 14:12:29.158973 5197 raft_consensus.cc:2932] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } } }
I20240503 14:12:29.163733 3128 catalog_manager.cc:5484] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index -1 to 4, NON_VOTER 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:29.166251 3206 consensus_peers.cc:485] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d -> Peer 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6:36477): Couldn't send request to peer 6c12721557f44ff28c4be1bcef9244c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ce9f735e2044c41900129f35549e7f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.180135 3272 consensus_queue.cc:237] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: true } }
I20240503 14:12:29.182345 5197 raft_consensus.cc:2932] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: true } } }
I20240503 14:12:29.186865 5093 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
I20240503 14:12:29.188467 3128 catalog_manager.cc:5484] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index -1 to 4, NON_VOTER 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:29.189456 3206 consensus_peers.cc:485] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d -> Peer 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3:34149): Couldn't send request to peer 89194a0dee124ed1a36b1e81aba11e8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a78ff4930264840b7696ab966699f21. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.200982 3403 consensus_queue.cc:237] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } }
I20240503 14:12:29.202550 5183 raft_consensus.cc:2932] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } } }
W20240503 14:12:29.203151 3336 consensus_peers.cc:485] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d -> Peer b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5:39315): Couldn't send request to peer b10cbf74e3c34a6c8c1972c44e9d3d40. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c67b31603ba3418c99839bf4d9cc6eb0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.207175 3128 catalog_manager.cc:5484] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index -1 to 4, NON_VOTER b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:29.218632 3272 consensus_queue.cc:237] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } }
I20240503 14:12:29.220490 5196 raft_consensus.cc:2932] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } } }
W20240503 14:12:29.220795 3208 consensus_peers.cc:485] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d -> Peer ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4:39891): Couldn't send request to peer ed03e4970b2844df8fb33aa1a1b440c9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9dca255e05f4408e82caa70e2dc72af6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.225426 3128 catalog_manager.cc:5484] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index -1 to 4, NON_VOTER ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:29.241179 3272 consensus_queue.cc:237] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } }
I20240503 14:12:29.243350 5197 raft_consensus.cc:2932] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } } }
I20240503 14:12:29.247891 3131 catalog_manager.cc:5484] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index -1 to 4, NON_VOTER f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:29.249984 3206 consensus_peers.cc:485] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d -> Peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7:43489): Couldn't send request to peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 291abe2ff8214a3bac8cb9c98b8b20c2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.261359 3403 consensus_queue.cc:237] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } }
I20240503 14:12:29.263235 5183 raft_consensus.cc:2932] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } } }
I20240503 14:12:29.269124 3131 catalog_manager.cc:5484] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index -1 to 4, NON_VOTER 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:29.270274 3337 consensus_peers.cc:485] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d -> Peer 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6:36477): Couldn't send request to peer 6c12721557f44ff28c4be1bcef9244c6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67f73e79bf0344bf9bbc2b1ecf3196a5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:29.562036 5286 ts_tablet_manager.cc:902] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: Initiating tablet copy from peer f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:29.563822 5286 tablet_copy_client.cc:308] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:46751
I20240503 14:12:29.572022 3292 tablet_copy_service.cc:140] P f7f1ddc70d7f442296eb038048b1069d: Received BeginTabletCopySession request for tablet 5a78ff4930264840b7696ab966699f21 from peer 89194a0dee124ed1a36b1e81aba11e8c ({username='slave'} at 127.0.186.3:36339)
I20240503 14:12:29.572311 3292 tablet_copy_service.cc:161] P f7f1ddc70d7f442296eb038048b1069d: Beginning new tablet copy session on tablet 5a78ff4930264840b7696ab966699f21 from peer 89194a0dee124ed1a36b1e81aba11e8c at {username='slave'} at 127.0.186.3:36339: session id = 89194a0dee124ed1a36b1e81aba11e8c-5a78ff4930264840b7696ab966699f21
I20240503 14:12:29.575122 3292 tablet_copy_source_session.cc:205] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.578487 5286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a78ff4930264840b7696ab966699f21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.584707 5301 ts_tablet_manager.cc:902] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Initiating tablet copy from peer fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:29.586696 5286 tablet_copy_client.cc:783] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.587105 5286 tablet_copy_client.cc:647] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.589883 5301 tablet_copy_client.cc:308] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:41985
I20240503 14:12:29.590226 5286 tablet_copy_client.cc:523] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.593847 5286 tablet_bootstrap.cc:492] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: Bootstrap starting.
I20240503 14:12:29.599417 3423 tablet_copy_service.cc:140] P fb0746b4d18f470f8815143973d6cf4d: Received BeginTabletCopySession request for tablet c09ddc167734479ca650e195092f4eee from peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9 ({username='slave'} at 127.0.186.7:45645)
I20240503 14:12:29.599865 3423 tablet_copy_service.cc:161] P fb0746b4d18f470f8815143973d6cf4d: Beginning new tablet copy session on tablet c09ddc167734479ca650e195092f4eee from peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9 at {username='slave'} at 127.0.186.7:45645: session id = f42dc7cc6eec4c1e9f6b6f7721b5d2d9-c09ddc167734479ca650e195092f4eee
I20240503 14:12:29.604899 3423 tablet_copy_source_session.cc:205] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.611508 5301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c09ddc167734479ca650e195092f4eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.626829 5301 tablet_copy_client.cc:783] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.627519 5301 tablet_copy_client.cc:647] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.630824 5301 tablet_copy_client.cc:523] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.635668 5301 tablet_bootstrap.cc:492] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Bootstrap starting.
I20240503 14:12:29.645306 5286 tablet_bootstrap.cc:492] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:29.645763 5286 tablet_bootstrap.cc:492] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: Bootstrap complete.
I20240503 14:12:29.646077 5286 ts_tablet_manager.cc:1367] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: Time spent bootstrapping tablet: real 0.052s user 0.052s sys 0.001s
I20240503 14:12:29.647665 5301 log.cc:824] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:29.649081 5324 ts_tablet_manager.cc:902] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: Initiating tablet copy from peer fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:29.652498 5286 raft_consensus.cc:348] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: true } }
I20240503 14:12:29.654381 5286 raft_consensus.cc:729] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 89194a0dee124ed1a36b1e81aba11e8c, State: Initialized, Role: LEARNER
I20240503 14:12:29.654611 5286 consensus_queue.cc:260] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: true } }
I20240503 14:12:29.656005 5286 ts_tablet_manager.cc:1398] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:12:29.656996 3292 tablet_copy_service.cc:342] P f7f1ddc70d7f442296eb038048b1069d: Request end of tablet copy session 89194a0dee124ed1a36b1e81aba11e8c-5a78ff4930264840b7696ab966699f21 received from {username='slave'} at 127.0.186.3:36339
I20240503 14:12:29.657166 3292 tablet_copy_service.cc:433] P f7f1ddc70d7f442296eb038048b1069d: ending tablet copy session 89194a0dee124ed1a36b1e81aba11e8c-5a78ff4930264840b7696ab966699f21 on tablet 5a78ff4930264840b7696ab966699f21 with peer 89194a0dee124ed1a36b1e81aba11e8c
I20240503 14:12:29.667220 5325 ts_tablet_manager.cc:902] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: Initiating tablet copy from peer f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:29.672065 5330 ts_tablet_manager.cc:902] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: Initiating tablet copy from peer fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:29.673151 5325 tablet_copy_client.cc:308] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:46751
I20240503 14:12:29.691149 5343 ts_tablet_manager.cc:902] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: Initiating tablet copy from peer fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:29.692382 5324 tablet_copy_client.cc:308] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:41985
I20240503 14:12:29.708778 3292 tablet_copy_service.cc:140] P f7f1ddc70d7f442296eb038048b1069d: Received BeginTabletCopySession request for tablet 2ce9f735e2044c41900129f35549e7f4 from peer 6c12721557f44ff28c4be1bcef9244c6 ({username='slave'} at 127.0.186.6:42515)
I20240503 14:12:29.709031 3292 tablet_copy_service.cc:161] P f7f1ddc70d7f442296eb038048b1069d: Beginning new tablet copy session on tablet 2ce9f735e2044c41900129f35549e7f4 from peer 6c12721557f44ff28c4be1bcef9244c6 at {username='slave'} at 127.0.186.6:42515: session id = 6c12721557f44ff28c4be1bcef9244c6-2ce9f735e2044c41900129f35549e7f4
I20240503 14:12:29.711997 5343 tablet_copy_client.cc:308] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:41985
I20240503 14:12:29.712139 3292 tablet_copy_source_session.cc:205] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.722410 5325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ce9f735e2044c41900129f35549e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.737380 5301 tablet_bootstrap.cc:492] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:29.738371 5301 tablet_bootstrap.cc:492] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Bootstrap complete.
I20240503 14:12:29.739135 5301 ts_tablet_manager.cc:1367] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Time spent bootstrapping tablet: real 0.104s user 0.063s sys 0.021s
I20240503 14:12:29.748641 5301 raft_consensus.cc:348] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } }
I20240503 14:12:29.749497 5301 raft_consensus.cc:729] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f42dc7cc6eec4c1e9f6b6f7721b5d2d9, State: Initialized, Role: LEARNER
I20240503 14:12:29.750437 5356 ts_tablet_manager.cc:902] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Initiating tablet copy from peer f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:29.750478 5301 consensus_queue.cc:260] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } }
I20240503 14:12:29.763568 5355 ts_tablet_manager.cc:902] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: Initiating tablet copy from peer f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:29.766213 5357 ts_tablet_manager.cc:902] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: Initiating tablet copy from peer f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:29.778414 5301 ts_tablet_manager.cc:1398] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Time spent starting tablet: real 0.039s user 0.015s sys 0.004s
I20240503 14:12:29.780762 5356 tablet_copy_client.cc:308] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:46751
I20240503 14:12:29.792902 3292 tablet_copy_service.cc:140] P f7f1ddc70d7f442296eb038048b1069d: Received BeginTabletCopySession request for tablet 291abe2ff8214a3bac8cb9c98b8b20c2 from peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9 ({username='slave'} at 127.0.186.7:41867)
I20240503 14:12:29.793114 3292 tablet_copy_service.cc:161] P f7f1ddc70d7f442296eb038048b1069d: Beginning new tablet copy session on tablet 291abe2ff8214a3bac8cb9c98b8b20c2 from peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9 at {username='slave'} at 127.0.186.7:41867: session id = f42dc7cc6eec4c1e9f6b6f7721b5d2d9-291abe2ff8214a3bac8cb9c98b8b20c2
I20240503 14:12:29.795404 3292 tablet_copy_source_session.cc:205] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.796856 3423 tablet_copy_service.cc:140] P fb0746b4d18f470f8815143973d6cf4d: Received BeginTabletCopySession request for tablet 67f73e79bf0344bf9bbc2b1ecf3196a5 from peer 6c12721557f44ff28c4be1bcef9244c6 ({username='slave'} at 127.0.186.6:46005)
I20240503 14:12:29.797125 3423 tablet_copy_service.cc:161] P fb0746b4d18f470f8815143973d6cf4d: Beginning new tablet copy session on tablet 67f73e79bf0344bf9bbc2b1ecf3196a5 from peer 6c12721557f44ff28c4be1bcef9244c6 at {username='slave'} at 127.0.186.6:46005: session id = 6c12721557f44ff28c4be1bcef9244c6-67f73e79bf0344bf9bbc2b1ecf3196a5
I20240503 14:12:29.797506 5356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 291abe2ff8214a3bac8cb9c98b8b20c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.801895 3423 tablet_copy_source_session.cc:205] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.804112 5324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67f73e79bf0344bf9bbc2b1ecf3196a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.807988 5356 tablet_copy_client.cc:783] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.808451 5356 tablet_copy_client.cc:647] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.810839 5356 tablet_copy_client.cc:523] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.816370 5356 tablet_bootstrap.cc:492] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Bootstrap starting.
I20240503 14:12:29.818645 3423 tablet_copy_service.cc:342] P fb0746b4d18f470f8815143973d6cf4d: Request end of tablet copy session f42dc7cc6eec4c1e9f6b6f7721b5d2d9-c09ddc167734479ca650e195092f4eee received from {username='slave'} at 127.0.186.7:45645
I20240503 14:12:29.818816 3423 tablet_copy_service.cc:433] P fb0746b4d18f470f8815143973d6cf4d: ending tablet copy session f42dc7cc6eec4c1e9f6b6f7721b5d2d9-c09ddc167734479ca650e195092f4eee on tablet c09ddc167734479ca650e195092f4eee with peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9
I20240503 14:12:29.823055 5324 tablet_copy_client.cc:783] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.832474 3423 tablet_copy_service.cc:140] P fb0746b4d18f470f8815143973d6cf4d: Received BeginTabletCopySession request for tablet d269db92e59e400db4db334fe177c4e6 from peer b10cbf74e3c34a6c8c1972c44e9d3d40 ({username='slave'} at 127.0.186.5:37441)
I20240503 14:12:29.832741 3423 tablet_copy_service.cc:161] P fb0746b4d18f470f8815143973d6cf4d: Beginning new tablet copy session on tablet d269db92e59e400db4db334fe177c4e6 from peer b10cbf74e3c34a6c8c1972c44e9d3d40 at {username='slave'} at 127.0.186.5:37441: session id = b10cbf74e3c34a6c8c1972c44e9d3d40-d269db92e59e400db4db334fe177c4e6
I20240503 14:12:29.836850 3423 tablet_copy_source_session.cc:205] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.843683 5324 tablet_copy_client.cc:647] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.844348 5343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d269db92e59e400db4db334fe177c4e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.846174 5325 tablet_copy_client.cc:783] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.846419 5325 tablet_copy_client.cc:647] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.852707 5330 tablet_copy_client.cc:308] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:41985
I20240503 14:12:29.853826 3423 tablet_copy_service.cc:140] P fb0746b4d18f470f8815143973d6cf4d: Received BeginTabletCopySession request for tablet c67b31603ba3418c99839bf4d9cc6eb0 from peer b10cbf74e3c34a6c8c1972c44e9d3d40 ({username='slave'} at 127.0.186.5:37441)
I20240503 14:12:29.854044 3423 tablet_copy_service.cc:161] P fb0746b4d18f470f8815143973d6cf4d: Beginning new tablet copy session on tablet c67b31603ba3418c99839bf4d9cc6eb0 from peer b10cbf74e3c34a6c8c1972c44e9d3d40 at {username='slave'} at 127.0.186.5:37441: session id = b10cbf74e3c34a6c8c1972c44e9d3d40-c67b31603ba3418c99839bf4d9cc6eb0
I20240503 14:12:29.858589 3423 tablet_copy_source_session.cc:205] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.859457 5325 tablet_copy_client.cc:523] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.860399 5330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c67b31603ba3418c99839bf4d9cc6eb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.863998 5324 tablet_copy_client.cc:523] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.867794 5343 tablet_copy_client.cc:783] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.868129 5343 tablet_copy_client.cc:647] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.868384 5330 tablet_copy_client.cc:783] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.868613 5330 tablet_copy_client.cc:647] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.871614 5343 tablet_copy_client.cc:523] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.876994 5343 tablet_bootstrap.cc:492] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: Bootstrap starting.
I20240503 14:12:29.878500 5325 tablet_bootstrap.cc:492] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: Bootstrap starting.
I20240503 14:12:29.880080 5330 tablet_copy_client.cc:523] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.880792 5355 tablet_copy_client.cc:308] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:46751
I20240503 14:12:29.879498 5324 tablet_bootstrap.cc:492] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: Bootstrap starting.
I20240503 14:12:29.883775 5330 tablet_bootstrap.cc:492] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: Bootstrap starting.
I20240503 14:12:29.889326 5357 tablet_copy_client.cc:308] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:46751
I20240503 14:12:29.891623 5330 log.cc:824] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:29.906728 5356 tablet_bootstrap.cc:492] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:29.907235 5356 tablet_bootstrap.cc:492] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Bootstrap complete.
I20240503 14:12:29.907545 5356 ts_tablet_manager.cc:1367] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Time spent bootstrapping tablet: real 0.091s user 0.063s sys 0.009s
I20240503 14:12:29.908609 5356 raft_consensus.cc:348] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } }
I20240503 14:12:29.908880 5356 raft_consensus.cc:729] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f42dc7cc6eec4c1e9f6b6f7721b5d2d9, State: Initialized, Role: LEARNER
I20240503 14:12:29.909152 5356 consensus_queue.cc:260] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: true } }
I20240503 14:12:29.910425 5356 ts_tablet_manager.cc:1398] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:29.912104 3292 tablet_copy_service.cc:342] P f7f1ddc70d7f442296eb038048b1069d: Request end of tablet copy session f42dc7cc6eec4c1e9f6b6f7721b5d2d9-291abe2ff8214a3bac8cb9c98b8b20c2 received from {username='slave'} at 127.0.186.7:41867
I20240503 14:12:29.912269 3292 tablet_copy_service.cc:433] P f7f1ddc70d7f442296eb038048b1069d: ending tablet copy session f42dc7cc6eec4c1e9f6b6f7721b5d2d9-291abe2ff8214a3bac8cb9c98b8b20c2 on tablet 291abe2ff8214a3bac8cb9c98b8b20c2 with peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9
I20240503 14:12:29.915273 3292 tablet_copy_service.cc:140] P f7f1ddc70d7f442296eb038048b1069d: Received BeginTabletCopySession request for tablet b6b48c2a49e249348b2a8c5109d251ce from peer ed03e4970b2844df8fb33aa1a1b440c9 ({username='slave'} at 127.0.186.4:43061)
I20240503 14:12:29.915535 3292 tablet_copy_service.cc:161] P f7f1ddc70d7f442296eb038048b1069d: Beginning new tablet copy session on tablet b6b48c2a49e249348b2a8c5109d251ce from peer ed03e4970b2844df8fb33aa1a1b440c9 at {username='slave'} at 127.0.186.4:43061: session id = ed03e4970b2844df8fb33aa1a1b440c9-b6b48c2a49e249348b2a8c5109d251ce
I20240503 14:12:29.915273 3291 tablet_copy_service.cc:140] P f7f1ddc70d7f442296eb038048b1069d: Received BeginTabletCopySession request for tablet 9dca255e05f4408e82caa70e2dc72af6 from peer ed03e4970b2844df8fb33aa1a1b440c9 ({username='slave'} at 127.0.186.4:43061)
I20240503 14:12:29.920174 3291 tablet_copy_service.cc:161] P f7f1ddc70d7f442296eb038048b1069d: Beginning new tablet copy session on tablet 9dca255e05f4408e82caa70e2dc72af6 from peer ed03e4970b2844df8fb33aa1a1b440c9 at {username='slave'} at 127.0.186.4:43061: session id = ed03e4970b2844df8fb33aa1a1b440c9-9dca255e05f4408e82caa70e2dc72af6
I20240503 14:12:29.923398 3291 tablet_copy_source_session.cc:205] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.924039 3292 tablet_copy_source_session.cc:205] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:29.929246 5357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dca255e05f4408e82caa70e2dc72af6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.929824 5355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b48c2a49e249348b2a8c5109d251ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:29.942083 5324 log.cc:824] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:29.946380 5357 tablet_copy_client.cc:783] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.946820 5357 tablet_copy_client.cc:647] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.949047 5355 tablet_copy_client.cc:783] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:29.949313 5357 tablet_copy_client.cc:523] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.949316 5355 tablet_copy_client.cc:647] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:29.951354 5355 tablet_copy_client.cc:523] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:29.955541 5357 tablet_bootstrap.cc:492] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: Bootstrap starting.
I20240503 14:12:29.958350 5355 tablet_bootstrap.cc:492] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: Bootstrap starting.
I20240503 14:12:29.982200 5330 tablet_bootstrap.cc:492] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:29.982297 5343 tablet_bootstrap.cc:492] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:29.983093 5343 tablet_bootstrap.cc:492] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: Bootstrap complete.
I20240503 14:12:29.983117 5330 tablet_bootstrap.cc:492] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: Bootstrap complete.
I20240503 14:12:29.983683 5330 ts_tablet_manager.cc:1367] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: Time spent bootstrapping tablet: real 0.100s user 0.049s sys 0.013s
I20240503 14:12:29.983691 5343 ts_tablet_manager.cc:1367] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: Time spent bootstrapping tablet: real 0.107s user 0.066s sys 0.005s
I20240503 14:12:29.988536 5357 log.cc:824] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:29.990788 5343 raft_consensus.cc:348] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } }
I20240503 14:12:29.990796 5330 raft_consensus.cc:348] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } }
I20240503 14:12:29.991473 5343 raft_consensus.cc:729] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b10cbf74e3c34a6c8c1972c44e9d3d40, State: Initialized, Role: LEARNER
I20240503 14:12:29.991534 5330 raft_consensus.cc:729] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b10cbf74e3c34a6c8c1972c44e9d3d40, State: Initialized, Role: LEARNER
I20240503 14:12:29.992205 5343 consensus_queue.cc:260] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } }
I20240503 14:12:29.992240 5330 consensus_queue.cc:260] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: true } }
I20240503 14:12:30.002341 5330 ts_tablet_manager.cc:1398] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40: Time spent starting tablet: real 0.018s user 0.007s sys 0.008s
I20240503 14:12:30.008951 3422 tablet_copy_service.cc:342] P fb0746b4d18f470f8815143973d6cf4d: Request end of tablet copy session b10cbf74e3c34a6c8c1972c44e9d3d40-c67b31603ba3418c99839bf4d9cc6eb0 received from {username='slave'} at 127.0.186.5:37441
I20240503 14:12:30.009150 3422 tablet_copy_service.cc:433] P fb0746b4d18f470f8815143973d6cf4d: ending tablet copy session b10cbf74e3c34a6c8c1972c44e9d3d40-c67b31603ba3418c99839bf4d9cc6eb0 on tablet c67b31603ba3418c99839bf4d9cc6eb0 with peer b10cbf74e3c34a6c8c1972c44e9d3d40
I20240503 14:12:30.016402 5343 ts_tablet_manager.cc:1398] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40: Time spent starting tablet: real 0.032s user 0.004s sys 0.008s
I20240503 14:12:30.018023 3422 tablet_copy_service.cc:342] P fb0746b4d18f470f8815143973d6cf4d: Request end of tablet copy session b10cbf74e3c34a6c8c1972c44e9d3d40-d269db92e59e400db4db334fe177c4e6 received from {username='slave'} at 127.0.186.5:37441
I20240503 14:12:30.018175 3422 tablet_copy_service.cc:433] P fb0746b4d18f470f8815143973d6cf4d: ending tablet copy session b10cbf74e3c34a6c8c1972c44e9d3d40-d269db92e59e400db4db334fe177c4e6 on tablet d269db92e59e400db4db334fe177c4e6 with peer b10cbf74e3c34a6c8c1972c44e9d3d40
I20240503 14:12:30.023382 5325 tablet_bootstrap.cc:492] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:30.024006 5325 tablet_bootstrap.cc:492] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: Bootstrap complete.
I20240503 14:12:30.024533 5325 ts_tablet_manager.cc:1367] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: Time spent bootstrapping tablet: real 0.146s user 0.059s sys 0.010s
I20240503 14:12:30.031651 5325 raft_consensus.cc:348] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } }
I20240503 14:12:30.032215 5325 raft_consensus.cc:729] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c12721557f44ff28c4be1bcef9244c6, State: Initialized, Role: LEARNER
I20240503 14:12:30.032826 5325 consensus_queue.cc:260] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } }
I20240503 14:12:30.041302 5325 ts_tablet_manager.cc:1398] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6: Time spent starting tablet: real 0.017s user 0.010s sys 0.006s
I20240503 14:12:30.049883 5324 tablet_bootstrap.cc:492] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:30.050697 5324 tablet_bootstrap.cc:492] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: Bootstrap complete.
I20240503 14:12:30.051162 5324 ts_tablet_manager.cc:1367] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: Time spent bootstrapping tablet: real 0.172s user 0.068s sys 0.000s
I20240503 14:12:30.052549 3292 tablet_copy_service.cc:342] P f7f1ddc70d7f442296eb038048b1069d: Request end of tablet copy session 6c12721557f44ff28c4be1bcef9244c6-2ce9f735e2044c41900129f35549e7f4 received from {username='slave'} at 127.0.186.6:42515
I20240503 14:12:30.052702 3292 tablet_copy_service.cc:433] P f7f1ddc70d7f442296eb038048b1069d: ending tablet copy session 6c12721557f44ff28c4be1bcef9244c6-2ce9f735e2044c41900129f35549e7f4 on tablet 2ce9f735e2044c41900129f35549e7f4 with peer 6c12721557f44ff28c4be1bcef9244c6
I20240503 14:12:30.052850 5324 raft_consensus.cc:348] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } }
I20240503 14:12:30.053165 5324 raft_consensus.cc:729] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c12721557f44ff28c4be1bcef9244c6, State: Initialized, Role: LEARNER
I20240503 14:12:30.053668 5324 consensus_queue.cc:260] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: true } }
I20240503 14:12:30.064990 4918 raft_consensus.cc:1203] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.079739 5324 ts_tablet_manager.cc:1398] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6: Time spent starting tablet: real 0.028s user 0.015s sys 0.012s
I20240503 14:12:30.081187 3422 tablet_copy_service.cc:342] P fb0746b4d18f470f8815143973d6cf4d: Request end of tablet copy session 6c12721557f44ff28c4be1bcef9244c6-67f73e79bf0344bf9bbc2b1ecf3196a5 received from {username='slave'} at 127.0.186.6:46005
I20240503 14:12:30.081337 3422 tablet_copy_service.cc:433] P fb0746b4d18f470f8815143973d6cf4d: ending tablet copy session 6c12721557f44ff28c4be1bcef9244c6-67f73e79bf0344bf9bbc2b1ecf3196a5 on tablet 67f73e79bf0344bf9bbc2b1ecf3196a5 with peer 6c12721557f44ff28c4be1bcef9244c6
I20240503 14:12:30.091281 5355 tablet_bootstrap.cc:492] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:30.092005 5355 tablet_bootstrap.cc:492] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: Bootstrap complete.
I20240503 14:12:30.092629 5355 ts_tablet_manager.cc:1367] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: Time spent bootstrapping tablet: real 0.134s user 0.046s sys 0.018s
I20240503 14:12:30.093746 4788 raft_consensus.cc:1203] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.096426 5357 tablet_bootstrap.cc:492] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:30.097041 5357 tablet_bootstrap.cc:492] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: Bootstrap complete.
I20240503 14:12:30.097411 5357 ts_tablet_manager.cc:1367] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: Time spent bootstrapping tablet: real 0.142s user 0.063s sys 0.012s
I20240503 14:12:30.099820 5355 raft_consensus.cc:348] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } }
I20240503 14:12:30.099831 5357 raft_consensus.cc:348] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } }
I20240503 14:12:30.100519 5355 raft_consensus.cc:729] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed03e4970b2844df8fb33aa1a1b440c9, State: Initialized, Role: LEARNER
I20240503 14:12:30.100639 5357 raft_consensus.cc:729] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ed03e4970b2844df8fb33aa1a1b440c9, State: Initialized, Role: LEARNER
I20240503 14:12:30.101161 5357 consensus_queue.cc:260] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } }
I20240503 14:12:30.101161 5355 consensus_queue.cc:260] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: true } }
I20240503 14:12:30.103389 5357 ts_tablet_manager.cc:1398] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240503 14:12:30.104139 5355 ts_tablet_manager.cc:1398] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9: Time spent starting tablet: real 0.011s user 0.009s sys 0.003s
I20240503 14:12:30.104423 3292 tablet_copy_service.cc:342] P f7f1ddc70d7f442296eb038048b1069d: Request end of tablet copy session ed03e4970b2844df8fb33aa1a1b440c9-9dca255e05f4408e82caa70e2dc72af6 received from {username='slave'} at 127.0.186.4:43061
I20240503 14:12:30.104584 3292 tablet_copy_service.cc:433] P f7f1ddc70d7f442296eb038048b1069d: ending tablet copy session ed03e4970b2844df8fb33aa1a1b440c9-9dca255e05f4408e82caa70e2dc72af6 on tablet 9dca255e05f4408e82caa70e2dc72af6 with peer ed03e4970b2844df8fb33aa1a1b440c9
I20240503 14:12:30.104911 3291 tablet_copy_service.cc:342] P f7f1ddc70d7f442296eb038048b1069d: Request end of tablet copy session ed03e4970b2844df8fb33aa1a1b440c9-b6b48c2a49e249348b2a8c5109d251ce received from {username='slave'} at 127.0.186.4:43061
I20240503 14:12:30.105044 3291 tablet_copy_service.cc:433] P f7f1ddc70d7f442296eb038048b1069d: ending tablet copy session ed03e4970b2844df8fb33aa1a1b440c9-b6b48c2a49e249348b2a8c5109d251ce on tablet b6b48c2a49e249348b2a8c5109d251ce with peer ed03e4970b2844df8fb33aa1a1b440c9
I20240503 14:12:30.113919 5048 raft_consensus.cc:1203] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.144256 4518 raft_consensus.cc:1203] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.166307 4657 raft_consensus.cc:1203] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.175052 4918 raft_consensus.cc:1203] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.246415 5048 raft_consensus.cc:1203] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.248531 4788 raft_consensus.cc:1203] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.323098 4657 raft_consensus.cc:1203] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:30.563849 5401 raft_consensus.cc:1050] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: attempting to promote NON_VOTER b10cbf74e3c34a6c8c1972c44e9d3d40 to VOTER
I20240503 14:12:30.564680 5401 consensus_queue.cc:237] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:30.566927 4788 raft_consensus.cc:1261] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 LEARNER]: Refusing update from remote peer fb0746b4d18f470f8815143973d6cf4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.567667 5183 consensus_queue.cc:1035] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:30.571420 5401 raft_consensus.cc:2932] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } } }
I20240503 14:12:30.572310 4788 raft_consensus.cc:2932] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } } }
I20240503 14:12:30.576758 3128 catalog_manager.cc:5484] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index 4 to 5, b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:30.625905 5183 raft_consensus.cc:1050] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: attempting to promote NON_VOTER f42dc7cc6eec4c1e9f6b6f7721b5d2d9 to VOTER
I20240503 14:12:30.626663 5183 consensus_queue.cc:237] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:30.637159 5048 raft_consensus.cc:1261] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 LEARNER]: Refusing update from remote peer fb0746b4d18f470f8815143973d6cf4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.637782 5401 consensus_queue.cc:1035] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:30.641496 5183 raft_consensus.cc:2932] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } } }
I20240503 14:12:30.642295 5048 raft_consensus.cc:2932] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } } }
I20240503 14:12:30.646423 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d269db92e59e400db4db334fe177c4e6"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58534
I20240503 14:12:30.646689 3403 raft_consensus.cc:595] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:30.647151 3131 catalog_manager.cc:5484] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index 4 to 5, f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:30.666515 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c09ddc167734479ca650e195092f4eee"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58554
I20240503 14:12:30.666771 3403 raft_consensus.cc:595] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:30.674284 5404 raft_consensus.cc:1050] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: attempting to promote NON_VOTER ed03e4970b2844df8fb33aa1a1b440c9 to VOTER
I20240503 14:12:30.675161 5404 consensus_queue.cc:237] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:30.677661 4657 raft_consensus.cc:1261] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 LEARNER]: Refusing update from remote peer f7f1ddc70d7f442296eb038048b1069d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.678812 5397 consensus_queue.cc:1035] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:30.683336 5404 raft_consensus.cc:2932] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } } }
I20240503 14:12:30.685388 4657 raft_consensus.cc:2932] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } } }
I20240503 14:12:30.685936 5397 raft_consensus.cc:1050] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: attempting to promote NON_VOTER 6c12721557f44ff28c4be1bcef9244c6 to VOTER
I20240503 14:12:30.686450 5397 consensus_queue.cc:237] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:30.688349 4918 raft_consensus.cc:1261] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 LEARNER]: Refusing update from remote peer f7f1ddc70d7f442296eb038048b1069d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.689163 5197 consensus_queue.cc:1035] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:30.690716 3128 catalog_manager.cc:5484] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 reported cstate change: config changed from index 4 to 5, ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } } }
I20240503 14:12:30.701391 5197 raft_consensus.cc:2932] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } } }
I20240503 14:12:30.702311 4918 raft_consensus.cc:2932] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } } }
I20240503 14:12:30.705790 3131 catalog_manager.cc:5484] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index 4 to 5, 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:30.729887 5404 raft_consensus.cc:1050] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: attempting to promote NON_VOTER 89194a0dee124ed1a36b1e81aba11e8c to VOTER
I20240503 14:12:30.730679 5404 consensus_queue.cc:237] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } }
I20240503 14:12:30.733227 4518 raft_consensus.cc:1261] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 LEARNER]: Refusing update from remote peer f7f1ddc70d7f442296eb038048b1069d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.733867 5397 consensus_queue.cc:1035] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Connected to new peer: Peer: permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:30.737591 4518 raft_consensus.cc:2932] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } } }
I20240503 14:12:30.739375 5400 raft_consensus.cc:1050] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: attempting to promote NON_VOTER b10cbf74e3c34a6c8c1972c44e9d3d40 to VOTER
I20240503 14:12:30.736898 5404 raft_consensus.cc:2932] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } } }
I20240503 14:12:30.740072 5400 consensus_queue.cc:237] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:30.742059 4788 raft_consensus.cc:1261] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 LEARNER]: Refusing update from remote peer fb0746b4d18f470f8815143973d6cf4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.742805 5401 consensus_queue.cc:1035] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:30.743575 3128 catalog_manager.cc:5484] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c reported cstate change: config changed from index 4 to 5, 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } } }
I20240503 14:12:30.745671 5400 raft_consensus.cc:2932] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } } }
I20240503 14:12:30.746340 4788 raft_consensus.cc:2932] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } } }
I20240503 14:12:30.749110 5401 raft_consensus.cc:1050] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: attempting to promote NON_VOTER 6c12721557f44ff28c4be1bcef9244c6 to VOTER
I20240503 14:12:30.749838 5401 consensus_queue.cc:237] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:30.752238 3131 catalog_manager.cc:5484] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 reported cstate change: config changed from index 4 to 5, b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } } }
I20240503 14:12:30.758307 4918 raft_consensus.cc:1261] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 LEARNER]: Refusing update from remote peer fb0746b4d18f470f8815143973d6cf4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.759090 5183 consensus_queue.cc:1035] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:30.762315 5401 raft_consensus.cc:2932] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } } }
I20240503 14:12:30.763051 4918 raft_consensus.cc:2932] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } } }
I20240503 14:12:30.767874 3128 catalog_manager.cc:5484] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index 4 to 5, 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:30.793628 5404 raft_consensus.cc:1050] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: attempting to promote NON_VOTER f42dc7cc6eec4c1e9f6b6f7721b5d2d9 to VOTER
I20240503 14:12:30.794297 5404 consensus_queue.cc:237] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:30.796259 5048 raft_consensus.cc:1261] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 LEARNER]: Refusing update from remote peer f7f1ddc70d7f442296eb038048b1069d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.796851 5397 consensus_queue.cc:1035] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:30.799428 5404 raft_consensus.cc:2932] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } } }
I20240503 14:12:30.800187 5048 raft_consensus.cc:2932] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } } }
I20240503 14:12:30.803854 3131 catalog_manager.cc:5484] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index 4 to 5, f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:30.911015 5404 raft_consensus.cc:1050] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: attempting to promote NON_VOTER ed03e4970b2844df8fb33aa1a1b440c9 to VOTER
I20240503 14:12:30.911765 5404 consensus_queue.cc:237] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:30.914412 4657 raft_consensus.cc:1261] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 LEARNER]: Refusing update from remote peer f7f1ddc70d7f442296eb038048b1069d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:30.915551 5404 consensus_queue.cc:1035] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 14:12:30.915829 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67f73e79bf0344bf9bbc2b1ecf3196a5"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58576
I20240503 14:12:30.916047 3403 raft_consensus.cc:595] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:30.926260 5397 raft_consensus.cc:2932] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } } }
I20240503 14:12:30.931128 3128 catalog_manager.cc:5484] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index 4 to 5, ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:30.930402 4657 raft_consensus.cc:2932] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } } }
I20240503 14:12:30.954934 3272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9dca255e05f4408e82caa70e2dc72af6"
dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52616
I20240503 14:12:30.955199 3272 raft_consensus.cc:595] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:30.972915 3272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "291abe2ff8214a3bac8cb9c98b8b20c2"
dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52628
I20240503 14:12:30.973176 3272 raft_consensus.cc:595] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:30.990877 3272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2ce9f735e2044c41900129f35549e7f4"
dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52650
I20240503 14:12:30.991107 3272 raft_consensus.cc:595] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:31.007800 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c67b31603ba3418c99839bf4d9cc6eb0"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58598
I20240503 14:12:31.008011 3403 raft_consensus.cc:595] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:31.026260 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d269db92e59e400db4db334fe177c4e6"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58610
I20240503 14:12:31.026476 3403 raft_consensus.cc:595] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:31.027390 5401 raft_consensus.cc:979] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: : Instructing follower f42dc7cc6eec4c1e9f6b6f7721b5d2d9 to start an election
I20240503 14:12:31.027570 5401 raft_consensus.cc:1067] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Signalling peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9 to start an election
I20240503 14:12:31.029039 5048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c09ddc167734479ca650e195092f4eee"
dest_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9"
from {username='slave'} at 127.0.186.2:45379
I20240503 14:12:31.029358 5048 raft_consensus.cc:482] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.029606 5048 raft_consensus.cc:3037] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.032250 5048 raft_consensus.cc:504] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:31.034332 5048 leader_election.cc:288] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [CANDIDATE]: Term 2 election: Requested vote from peers fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:31.035060 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c09ddc167734479ca650e195092f4eee" candidate_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
I20240503 14:12:31.035715 3403 raft_consensus.cc:3032] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.035818 3403 raft_consensus.cc:729] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:31.036159 3403 consensus_queue.cc:260] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:31.036583 3403 raft_consensus.cc:3037] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.039237 3403 raft_consensus.cc:2454] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f42dc7cc6eec4c1e9f6b6f7721b5d2d9 in term 2.
I20240503 14:12:31.039942 4983 leader_election.cc:302] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f42dc7cc6eec4c1e9f6b6f7721b5d2d9, fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:31.040323 5359 raft_consensus.cc:2781] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.040753 5359 raft_consensus.cc:686] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 2 LEADER]: Becoming Leader. State: Replica: f42dc7cc6eec4c1e9f6b6f7721b5d2d9, State: Running, Role: LEADER
I20240503 14:12:31.041317 5359 consensus_queue.cc:237] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:31.045778 3128 catalog_manager.cc:5484] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 reported cstate change: term changed from 1 to 2, leader changed from fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) to f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7). New cstate: current_term: 2 leader_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:12:31.059684 3116 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet c09ddc167734479ca650e195092f4eee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7:43489) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 14:12:31.064318 3272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b6b48c2a49e249348b2a8c5109d251ce"
dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52660
I20240503 14:12:31.064530 3272 raft_consensus.cc:595] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:31.081642 3272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5a78ff4930264840b7696ab966699f21"
dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52664
I20240503 14:12:31.081869 3272 raft_consensus.cc:595] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:31.115273 5197 raft_consensus.cc:979] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: : Instructing follower 6c12721557f44ff28c4be1bcef9244c6 to start an election
I20240503 14:12:31.115453 5197 raft_consensus.cc:1067] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Signalling peer 6c12721557f44ff28c4be1bcef9244c6 to start an election
I20240503 14:12:31.116415 4918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2ce9f735e2044c41900129f35549e7f4"
dest_uuid: "6c12721557f44ff28c4be1bcef9244c6"
from {username='slave'} at 127.0.186.1:58257
I20240503 14:12:31.116691 4918 raft_consensus.cc:482] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.116885 4918 raft_consensus.cc:3037] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.119019 4918 raft_consensus.cc:504] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:31.120443 4918 leader_election.cc:288] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [CANDIDATE]: Term 2 election: Requested vote from peers f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:31.121150 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ce9f735e2044c41900129f35549e7f4" candidate_uuid: "6c12721557f44ff28c4be1bcef9244c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
I20240503 14:12:31.121441 3272 raft_consensus.cc:3032] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.121533 3272 raft_consensus.cc:729] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:31.121843 3272 consensus_queue.cc:260] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:31.122150 3272 raft_consensus.cc:3037] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.124399 3272 raft_consensus.cc:2454] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c12721557f44ff28c4be1bcef9244c6 in term 2.
I20240503 14:12:31.125066 4852 leader_election.cc:302] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6c12721557f44ff28c4be1bcef9244c6, f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:31.125512 5517 raft_consensus.cc:2781] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.125958 5517 raft_consensus.cc:686] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 2 LEADER]: Becoming Leader. State: Replica: 6c12721557f44ff28c4be1bcef9244c6, State: Running, Role: LEADER
I20240503 14:12:31.126406 5517 consensus_queue.cc:237] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:31.129647 3128 catalog_manager.cc:5484] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 reported cstate change: term changed from 1 to 2, leader changed from f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) to 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6). New cstate: current_term: 2 leader_uuid: "6c12721557f44ff28c4be1bcef9244c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.153513 5524 raft_consensus.cc:979] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: : Instructing follower b10cbf74e3c34a6c8c1972c44e9d3d40 to start an election
I20240503 14:12:31.153720 5524 raft_consensus.cc:1067] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Signalling peer b10cbf74e3c34a6c8c1972c44e9d3d40 to start an election
I20240503 14:12:31.154526 4788 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d269db92e59e400db4db334fe177c4e6"
dest_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40"
from {username='slave'} at 127.0.186.2:48427
I20240503 14:12:31.154814 4788 raft_consensus.cc:482] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.155007 4788 raft_consensus.cc:3037] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.157119 4788 raft_consensus.cc:504] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:31.158382 4788 leader_election.cc:288] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [CANDIDATE]: Term 2 election: Requested vote from peers fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:31.158994 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d269db92e59e400db4db334fe177c4e6" candidate_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
I20240503 14:12:31.159173 3403 raft_consensus.cc:3032] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.159268 3403 raft_consensus.cc:729] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:31.159561 3403 consensus_queue.cc:260] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:31.159906 3403 raft_consensus.cc:3037] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.161849 3403 raft_consensus.cc:2454] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b10cbf74e3c34a6c8c1972c44e9d3d40 in term 2.
I20240503 14:12:31.162379 4723 leader_election.cc:302] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b10cbf74e3c34a6c8c1972c44e9d3d40, fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:31.162801 5389 raft_consensus.cc:2781] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.163235 5389 raft_consensus.cc:686] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 2 LEADER]: Becoming Leader. State: Replica: b10cbf74e3c34a6c8c1972c44e9d3d40, State: Running, Role: LEADER
I20240503 14:12:31.163805 5389 consensus_queue.cc:237] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:31.167894 3128 catalog_manager.cc:5484] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 reported cstate change: term changed from 1 to 2, leader changed from fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) to b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5). New cstate: current_term: 2 leader_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.195247 5401 raft_consensus.cc:979] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: : Instructing follower b10cbf74e3c34a6c8c1972c44e9d3d40 to start an election
I20240503 14:12:31.195426 5401 raft_consensus.cc:1067] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Signalling peer b10cbf74e3c34a6c8c1972c44e9d3d40 to start an election
I20240503 14:12:31.196141 4788 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c67b31603ba3418c99839bf4d9cc6eb0"
dest_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40"
from {username='slave'} at 127.0.186.2:48427
I20240503 14:12:31.196416 4788 raft_consensus.cc:482] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.196532 4788 raft_consensus.cc:3037] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.198426 4788 raft_consensus.cc:504] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:31.198932 4788 leader_election.cc:288] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [CANDIDATE]: Term 2 election: Requested vote from peers fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:31.199630 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c67b31603ba3418c99839bf4d9cc6eb0" candidate_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
I20240503 14:12:31.199829 3403 raft_consensus.cc:3032] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.199915 3403 raft_consensus.cc:729] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:31.200141 3403 consensus_queue.cc:260] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:31.200382 3403 raft_consensus.cc:3037] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.202183 3403 raft_consensus.cc:2454] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b10cbf74e3c34a6c8c1972c44e9d3d40 in term 2.
I20240503 14:12:31.202592 4723 leader_election.cc:302] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: b10cbf74e3c34a6c8c1972c44e9d3d40, fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:31.202870 5389 raft_consensus.cc:2781] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.203035 5389 raft_consensus.cc:686] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 2 LEADER]: Becoming Leader. State: Replica: b10cbf74e3c34a6c8c1972c44e9d3d40, State: Running, Role: LEADER
I20240503 14:12:31.203299 5389 consensus_queue.cc:237] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:31.205605 3128 catalog_manager.cc:5484] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 reported cstate change: term changed from 1 to 2, leader changed from fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) to b10cbf74e3c34a6c8c1972c44e9d3d40 (127.0.186.5). New cstate: current_term: 2 leader_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.283277 5397 raft_consensus.cc:979] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: : Instructing follower ed03e4970b2844df8fb33aa1a1b440c9 to start an election
I20240503 14:12:31.283505 5397 raft_consensus.cc:1067] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Signalling peer ed03e4970b2844df8fb33aa1a1b440c9 to start an election
I20240503 14:12:31.284448 4657 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b6b48c2a49e249348b2a8c5109d251ce"
dest_uuid: "ed03e4970b2844df8fb33aa1a1b440c9"
from {username='slave'} at 127.0.186.1:32969
I20240503 14:12:31.284758 4657 raft_consensus.cc:482] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.284955 4657 raft_consensus.cc:3037] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.287168 4657 raft_consensus.cc:504] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:31.288766 4657 leader_election.cc:288] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [CANDIDATE]: Term 2 election: Requested vote from peers f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:31.289357 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6b48c2a49e249348b2a8c5109d251ce" candidate_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
I20240503 14:12:31.289572 3272 raft_consensus.cc:3032] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.289647 3272 raft_consensus.cc:729] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:31.289916 3272 consensus_queue.cc:260] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:31.290199 3272 raft_consensus.cc:3037] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.292187 3272 raft_consensus.cc:2454] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed03e4970b2844df8fb33aa1a1b440c9 in term 2.
I20240503 14:12:31.292753 4592 leader_election.cc:302] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ed03e4970b2844df8fb33aa1a1b440c9, f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:31.293210 5402 raft_consensus.cc:2781] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.293807 5402 raft_consensus.cc:686] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 2 LEADER]: Becoming Leader. State: Replica: ed03e4970b2844df8fb33aa1a1b440c9, State: Running, Role: LEADER
I20240503 14:12:31.294312 5402 consensus_queue.cc:237] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:31.298512 3128 catalog_manager.cc:5484] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 reported cstate change: term changed from 1 to 2, leader changed from f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) to ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4). New cstate: current_term: 2 leader_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.304643 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "67f73e79bf0344bf9bbc2b1ecf3196a5"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58650
I20240503 14:12:31.304889 3403 raft_consensus.cc:595] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:31.320031 5397 raft_consensus.cc:979] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: : Instructing follower f42dc7cc6eec4c1e9f6b6f7721b5d2d9 to start an election
I20240503 14:12:31.320205 5397 raft_consensus.cc:1067] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Signalling peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9 to start an election
I20240503 14:12:31.320279 5401 raft_consensus.cc:979] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: : Instructing follower 6c12721557f44ff28c4be1bcef9244c6 to start an election
I20240503 14:12:31.320395 5401 raft_consensus.cc:1067] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Signalling peer 6c12721557f44ff28c4be1bcef9244c6 to start an election
I20240503 14:12:31.321098 4918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "67f73e79bf0344bf9bbc2b1ecf3196a5"
dest_uuid: "6c12721557f44ff28c4be1bcef9244c6"
from {username='slave'} at 127.0.186.2:43643
I20240503 14:12:31.321065 5048 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "291abe2ff8214a3bac8cb9c98b8b20c2"
dest_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9"
from {username='slave'} at 127.0.186.1:37199
I20240503 14:12:31.321318 4918 raft_consensus.cc:482] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.321319 5048 raft_consensus.cc:482] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.321444 4918 raft_consensus.cc:3037] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.321444 5048 raft_consensus.cc:3037] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.324539 5048 raft_consensus.cc:504] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:31.325547 5048 leader_election.cc:288] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [CANDIDATE]: Term 2 election: Requested vote from peers f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:31.326445 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "291abe2ff8214a3bac8cb9c98b8b20c2" candidate_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
I20240503 14:12:31.326690 3272 raft_consensus.cc:3032] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.326781 3272 raft_consensus.cc:729] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:31.326823 5404 raft_consensus.cc:979] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: : Instructing follower ed03e4970b2844df8fb33aa1a1b440c9 to start an election
I20240503 14:12:31.326736 4918 raft_consensus.cc:504] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:31.327005 5404 raft_consensus.cc:1067] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Signalling peer ed03e4970b2844df8fb33aa1a1b440c9 to start an election
I20240503 14:12:31.327090 3272 consensus_queue.cc:260] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:31.327394 3272 raft_consensus.cc:3037] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.327754 4658 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9dca255e05f4408e82caa70e2dc72af6"
dest_uuid: "ed03e4970b2844df8fb33aa1a1b440c9"
from {username='slave'} at 127.0.186.1:32969
I20240503 14:12:31.327944 4658 raft_consensus.cc:482] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.328074 4658 raft_consensus.cc:3037] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.330354 3272 raft_consensus.cc:2454] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f42dc7cc6eec4c1e9f6b6f7721b5d2d9 in term 2.
I20240503 14:12:31.330914 4982 leader_election.cc:302] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: f42dc7cc6eec4c1e9f6b6f7721b5d2d9, f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:31.330829 4658 raft_consensus.cc:504] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:31.331221 5359 raft_consensus.cc:2781] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.331406 5359 raft_consensus.cc:686] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 2 LEADER]: Becoming Leader. State: Replica: f42dc7cc6eec4c1e9f6b6f7721b5d2d9, State: Running, Role: LEADER
I20240503 14:12:31.331707 5359 consensus_queue.cc:237] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:31.332726 4658 leader_election.cc:288] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [CANDIDATE]: Term 2 election: Requested vote from peers f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:31.333336 4918 leader_election.cc:288] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [CANDIDATE]: Term 2 election: Requested vote from peers fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:31.333395 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dca255e05f4408e82caa70e2dc72af6" candidate_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
I20240503 14:12:31.333617 3272 raft_consensus.cc:3032] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.333705 3272 raft_consensus.cc:729] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:31.334059 3272 consensus_queue.cc:260] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:31.334383 3272 raft_consensus.cc:3037] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.336301 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67f73e79bf0344bf9bbc2b1ecf3196a5" candidate_uuid: "6c12721557f44ff28c4be1bcef9244c6" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
I20240503 14:12:31.336539 3403 raft_consensus.cc:3032] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.336632 3403 raft_consensus.cc:729] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:31.336907 3403 consensus_queue.cc:260] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:31.337249 3403 raft_consensus.cc:3037] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.337649 3272 raft_consensus.cc:2454] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed03e4970b2844df8fb33aa1a1b440c9 in term 2.
I20240503 14:12:31.338080 4592 leader_election.cc:302] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ed03e4970b2844df8fb33aa1a1b440c9, f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:31.338357 5402 raft_consensus.cc:2781] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.338526 5402 raft_consensus.cc:686] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 2 LEADER]: Becoming Leader. State: Replica: ed03e4970b2844df8fb33aa1a1b440c9, State: Running, Role: LEADER
I20240503 14:12:31.338794 5402 consensus_queue.cc:237] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:31.340055 3403 raft_consensus.cc:2454] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c12721557f44ff28c4be1bcef9244c6 in term 2.
I20240503 14:12:31.335055 3128 catalog_manager.cc:5484] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 reported cstate change: term changed from 1 to 2, leader changed from f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) to f42dc7cc6eec4c1e9f6b6f7721b5d2d9 (127.0.186.7). New cstate: current_term: 2 leader_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.341290 3131 catalog_manager.cc:5484] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 reported cstate change: term changed from 1 to 2, leader changed from f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) to ed03e4970b2844df8fb33aa1a1b440c9 (127.0.186.4). New cstate: current_term: 2 leader_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.343312 4853 leader_election.cc:302] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6c12721557f44ff28c4be1bcef9244c6, fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:31.343685 5517 raft_consensus.cc:2781] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.343850 5517 raft_consensus.cc:686] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 2 LEADER]: Becoming Leader. State: Replica: 6c12721557f44ff28c4be1bcef9244c6, State: Running, Role: LEADER
I20240503 14:12:31.344074 5517 consensus_queue.cc:237] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:31.347851 3272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9dca255e05f4408e82caa70e2dc72af6"
dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52676
I20240503 14:12:31.348079 3272 raft_consensus.cc:595] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Received request to transfer leadership
I20240503 14:12:31.348181 3272 raft_consensus.cc:603] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240503 14:12:31.352895 3128 catalog_manager.cc:5484] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 reported cstate change: term changed from 1 to 2, leader changed from fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) to 6c12721557f44ff28c4be1bcef9244c6 (127.0.186.6). New cstate: current_term: 2 leader_uuid: "6c12721557f44ff28c4be1bcef9244c6" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.357698 5397 raft_consensus.cc:979] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: : Instructing follower 89194a0dee124ed1a36b1e81aba11e8c to start an election
I20240503 14:12:31.357872 5397 raft_consensus.cc:1067] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Signalling peer 89194a0dee124ed1a36b1e81aba11e8c to start an election
I20240503 14:12:31.358669 4518 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5a78ff4930264840b7696ab966699f21"
dest_uuid: "89194a0dee124ed1a36b1e81aba11e8c"
from {username='slave'} at 127.0.186.1:35421
I20240503 14:12:31.358989 4518 raft_consensus.cc:482] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:31.359122 4518 raft_consensus.cc:3037] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.362264 4518 raft_consensus.cc:504] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } }
I20240503 14:12:31.363481 4518 leader_election.cc:288] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [CANDIDATE]: Term 2 election: Requested vote from peers f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:31.364229 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a78ff4930264840b7696ab966699f21" candidate_uuid: "89194a0dee124ed1a36b1e81aba11e8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
I20240503 14:12:31.364511 3272 raft_consensus.cc:3032] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:31.364630 3272 raft_consensus.cc:729] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:31.365065 3272 consensus_queue.cc:260] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } }
I20240503 14:12:31.365471 3272 raft_consensus.cc:3037] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:31.367825 3272 raft_consensus.cc:2454] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89194a0dee124ed1a36b1e81aba11e8c in term 2.
I20240503 14:12:31.368415 4450 leader_election.cc:302] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 89194a0dee124ed1a36b1e81aba11e8c, f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:31.369120 5650 raft_consensus.cc:2781] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:31.369560 5650 raft_consensus.cc:686] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 2 LEADER]: Becoming Leader. State: Replica: 89194a0dee124ed1a36b1e81aba11e8c, State: Running, Role: LEADER
I20240503 14:12:31.369887 5650 consensus_queue.cc:237] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } }
I20240503 14:12:31.373401 3128 catalog_manager.cc:5484] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c reported cstate change: term changed from 1 to 2, leader changed from f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) to 89194a0dee124ed1a36b1e81aba11e8c (127.0.186.3). New cstate: current_term: 2 leader_uuid: "89194a0dee124ed1a36b1e81aba11e8c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.462774 3403 raft_consensus.cc:1261] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Refusing update from remote peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.463817 5359 consensus_queue.cc:1035] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.627204 3403 raft_consensus.cc:1261] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Refusing update from remote peer b10cbf74e3c34a6c8c1972c44e9d3d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.627954 5389 consensus_queue.cc:1035] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.637634 3403 raft_consensus.cc:1261] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Refusing update from remote peer b10cbf74e3c34a6c8c1972c44e9d3d40: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.638279 5389 consensus_queue.cc:1035] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.667243 3272 raft_consensus.cc:1261] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Refusing update from remote peer 6c12721557f44ff28c4be1bcef9244c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.668279 5517 consensus_queue.cc:1035] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.708329 5048 consensus_queue.cc:237] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:31.709333 5359 raft_consensus.cc:2932] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } } }
I20240503 14:12:31.713057 3116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c09ddc167734479ca650e195092f4eee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:31.714905 3128 catalog_manager.cc:5484] T c09ddc167734479ca650e195092f4eee P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 reported cstate change: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.719369 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet c09ddc167734479ca650e195092f4eee with delete_type TABLET_DATA_TOMBSTONED (TS fb0746b4d18f470f8815143973d6cf4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33736
I20240503 14:12:31.720422 5724 tablet_replica.cc:330] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: stopping tablet replica
I20240503 14:12:31.720885 5724 raft_consensus.cc:2229] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.721132 5724 raft_consensus.cc:2258] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.722772 5724 ts_tablet_manager.cc:1875] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:31.729756 5724 ts_tablet_manager.cc:1888] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:31.729925 5724 log.cc:1196] T c09ddc167734479ca650e195092f4eee P fb0746b4d18f470f8815143973d6cf4d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/c09ddc167734479ca650e195092f4eee
I20240503 14:12:31.730798 3117 catalog_manager.cc:4830] TS fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985): tablet c09ddc167734479ca650e195092f4eee (table rebalance_test_table_1 [id=f2f3fdd1f0e04c8a8811861948f84daa]) successfully deleted
I20240503 14:12:31.765117 3272 raft_consensus.cc:1261] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Refusing update from remote peer ed03e4970b2844df8fb33aa1a1b440c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.765911 4918 consensus_queue.cc:237] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:31.765903 5402 consensus_queue.cc:1035] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.767182 5517 raft_consensus.cc:2932] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } } }
I20240503 14:12:31.772992 3116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2ce9f735e2044c41900129f35549e7f4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:31.775388 3128 catalog_manager.cc:5484] T 2ce9f735e2044c41900129f35549e7f4 P 6c12721557f44ff28c4be1bcef9244c6 reported cstate change: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New cstate: current_term: 2 leader_uuid: "6c12721557f44ff28c4be1bcef9244c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.780191 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ce9f735e2044c41900129f35549e7f4 with delete_type TABLET_DATA_TOMBSTONED (TS f7f1ddc70d7f442296eb038048b1069d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46464
I20240503 14:12:31.781049 5751 tablet_replica.cc:330] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: stopping tablet replica
I20240503 14:12:31.781536 5751 raft_consensus.cc:2229] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.781891 5751 raft_consensus.cc:2258] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.783571 5751 ts_tablet_manager.cc:1875] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:31.790185 5751 ts_tablet_manager.cc:1888] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:31.790361 5751 log.cc:1196] T 2ce9f735e2044c41900129f35549e7f4 P f7f1ddc70d7f442296eb038048b1069d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ce9f735e2044c41900129f35549e7f4
I20240503 14:12:31.791752 3116 catalog_manager.cc:4830] TS f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751): tablet 2ce9f735e2044c41900129f35549e7f4 (table rebalance_test_table_3 [id=a7ffaa5ef2a24eabac1144221dcd708c]) successfully deleted
I20240503 14:12:31.838618 3272 raft_consensus.cc:1261] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Refusing update from remote peer f42dc7cc6eec4c1e9f6b6f7721b5d2d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.839362 5359 consensus_queue.cc:1035] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.840901 4787 consensus_queue.cc:237] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:31.841924 5389 raft_consensus.cc:2932] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } } }
I20240503 14:12:31.846720 3272 raft_consensus.cc:1261] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Refusing update from remote peer 89194a0dee124ed1a36b1e81aba11e8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.846994 3115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d269db92e59e400db4db334fe177c4e6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:31.847775 5650 consensus_queue.cc:1035] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.847965 3128 catalog_manager.cc:5484] T d269db92e59e400db4db334fe177c4e6 P b10cbf74e3c34a6c8c1972c44e9d3d40 reported cstate change: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.851752 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet d269db92e59e400db4db334fe177c4e6 with delete_type TABLET_DATA_TOMBSTONED (TS fb0746b4d18f470f8815143973d6cf4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33736
I20240503 14:12:31.852581 5724 tablet_replica.cc:330] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: stopping tablet replica
I20240503 14:12:31.852825 5724 raft_consensus.cc:2229] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.853075 5724 raft_consensus.cc:2258] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.854154 5724 ts_tablet_manager.cc:1875] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:31.860085 5724 ts_tablet_manager.cc:1888] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:31.860226 5724 log.cc:1196] T d269db92e59e400db4db334fe177c4e6 P fb0746b4d18f470f8815143973d6cf4d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/d269db92e59e400db4db334fe177c4e6
I20240503 14:12:31.860971 3117 catalog_manager.cc:4830] TS fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985): tablet d269db92e59e400db4db334fe177c4e6 (table rebalance_test_table_3 [id=a7ffaa5ef2a24eabac1144221dcd708c]) successfully deleted
I20240503 14:12:31.864775 4787 consensus_queue.cc:237] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } }
I20240503 14:12:31.865617 5389 raft_consensus.cc:2932] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } } }
I20240503 14:12:31.868896 3115 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c67b31603ba3418c99839bf4d9cc6eb0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:31.870501 3128 catalog_manager.cc:5484] T c67b31603ba3418c99839bf4d9cc6eb0 P b10cbf74e3c34a6c8c1972c44e9d3d40 reported cstate change: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b10cbf74e3c34a6c8c1972c44e9d3d40" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 39315 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.875265 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet c67b31603ba3418c99839bf4d9cc6eb0 with delete_type TABLET_DATA_TOMBSTONED (TS fb0746b4d18f470f8815143973d6cf4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33736
I20240503 14:12:31.875682 5724 tablet_replica.cc:330] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: stopping tablet replica
I20240503 14:12:31.875880 5724 raft_consensus.cc:2229] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.876080 5724 raft_consensus.cc:2258] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.877161 5724 ts_tablet_manager.cc:1875] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:31.883009 3272 raft_consensus.cc:1261] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Refusing update from remote peer ed03e4970b2844df8fb33aa1a1b440c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.883724 5402 consensus_queue.cc:1035] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.883980 5724 ts_tablet_manager.cc:1888] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:31.884181 5724 log.cc:1196] T c67b31603ba3418c99839bf4d9cc6eb0 P fb0746b4d18f470f8815143973d6cf4d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/c67b31603ba3418c99839bf4d9cc6eb0
I20240503 14:12:31.885635 3117 catalog_manager.cc:4830] TS fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985): tablet c67b31603ba3418c99839bf4d9cc6eb0 (table rebalance_test_table_4 [id=50e14958ddc64d79a9d091ec5d86e98c]) successfully deleted
I20240503 14:12:31.912626 3403 raft_consensus.cc:1261] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Refusing update from remote peer 6c12721557f44ff28c4be1bcef9244c6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:31.913486 5517 consensus_queue.cc:1035] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:31.947110 4658 consensus_queue.cc:237] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:31.948160 5627 raft_consensus.cc:2932] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } } }
I20240503 14:12:31.952121 3118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6b48c2a49e249348b2a8c5109d251ce with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:31.953514 3131 catalog_manager.cc:5484] T b6b48c2a49e249348b2a8c5109d251ce P ed03e4970b2844df8fb33aa1a1b440c9 reported cstate change: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New cstate: current_term: 2 leader_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.957199 4658 consensus_queue.cc:237] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } }
I20240503 14:12:31.958420 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b48c2a49e249348b2a8c5109d251ce with delete_type TABLET_DATA_TOMBSTONED (TS f7f1ddc70d7f442296eb038048b1069d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46464
I20240503 14:12:31.958261 5402 raft_consensus.cc:2932] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } } }
I20240503 14:12:31.958773 5751 tablet_replica.cc:330] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: stopping tablet replica
I20240503 14:12:31.958935 5751 raft_consensus.cc:2229] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.959120 5751 raft_consensus.cc:2258] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.960332 5751 ts_tablet_manager.cc:1875] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:31.962271 3118 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9dca255e05f4408e82caa70e2dc72af6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:31.963027 5048 consensus_queue.cc:237] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } }
I20240503 14:12:31.964044 5595 raft_consensus.cc:2932] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } } }
I20240503 14:12:31.965168 3131 catalog_manager.cc:5484] T 9dca255e05f4408e82caa70e2dc72af6 P ed03e4970b2844df8fb33aa1a1b440c9 reported cstate change: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New cstate: current_term: 2 leader_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ed03e4970b2844df8fb33aa1a1b440c9" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 39891 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.968374 3116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 291abe2ff8214a3bac8cb9c98b8b20c2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:31.969463 3128 catalog_manager.cc:5484] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f42dc7cc6eec4c1e9f6b6f7721b5d2d9 reported cstate change: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New cstate: current_term: 2 leader_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f42dc7cc6eec4c1e9f6b6f7721b5d2d9" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 43489 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.972415 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dca255e05f4408e82caa70e2dc72af6 with delete_type TABLET_DATA_TOMBSTONED (TS f7f1ddc70d7f442296eb038048b1069d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46464
I20240503 14:12:31.972584 5751 ts_tablet_manager.cc:1888] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:31.972710 5751 log.cc:1196] T b6b48c2a49e249348b2a8c5109d251ce P f7f1ddc70d7f442296eb038048b1069d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/b6b48c2a49e249348b2a8c5109d251ce
I20240503 14:12:31.973552 5751 tablet_replica.cc:330] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: stopping tablet replica
I20240503 14:12:31.973587 3116 catalog_manager.cc:4830] TS f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751): tablet b6b48c2a49e249348b2a8c5109d251ce (table rebalance_test_table_4 [id=50e14958ddc64d79a9d091ec5d86e98c]) successfully deleted
I20240503 14:12:31.973716 5751 raft_consensus.cc:2229] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.973874 5751 raft_consensus.cc:2258] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.974997 5751 ts_tablet_manager.cc:1875] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:31.975430 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet 291abe2ff8214a3bac8cb9c98b8b20c2 with delete_type TABLET_DATA_TOMBSTONED (TS f7f1ddc70d7f442296eb038048b1069d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46464
I20240503 14:12:31.980311 5751 ts_tablet_manager.cc:1888] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:31.980448 5751 log.cc:1196] T 9dca255e05f4408e82caa70e2dc72af6 P f7f1ddc70d7f442296eb038048b1069d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/9dca255e05f4408e82caa70e2dc72af6
I20240503 14:12:31.981184 5751 tablet_replica.cc:330] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: stopping tablet replica
I20240503 14:12:31.981374 5751 raft_consensus.cc:2229] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.981339 3116 catalog_manager.cc:4830] TS f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751): tablet 9dca255e05f4408e82caa70e2dc72af6 (table rebalance_test_table_2 [id=c7b0ffc97dca41f2bc90a97e1631f1ec]) successfully deleted
I20240503 14:12:31.981578 5751 raft_consensus.cc:2258] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:31.982489 5751 ts_tablet_manager.cc:1875] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:31.986966 5751 ts_tablet_manager.cc:1888] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:31.987102 5751 log.cc:1196] T 291abe2ff8214a3bac8cb9c98b8b20c2 P f7f1ddc70d7f442296eb038048b1069d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/291abe2ff8214a3bac8cb9c98b8b20c2
I20240503 14:12:31.987871 3116 catalog_manager.cc:4830] TS f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751): tablet 291abe2ff8214a3bac8cb9c98b8b20c2 (table rebalance_test_table_2 [id=c7b0ffc97dca41f2bc90a97e1631f1ec]) successfully deleted
I20240503 14:12:31.988427 4918 consensus_queue.cc:237] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } }
I20240503 14:12:31.989281 5517 raft_consensus.cc:2932] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } } }
I20240503 14:12:31.992501 3116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67f73e79bf0344bf9bbc2b1ecf3196a5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:31.994350 3128 catalog_manager.cc:5484] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P 6c12721557f44ff28c4be1bcef9244c6 reported cstate change: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "6c12721557f44ff28c4be1bcef9244c6" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c12721557f44ff28c4be1bcef9244c6" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 36477 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:31.998478 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet 67f73e79bf0344bf9bbc2b1ecf3196a5 with delete_type TABLET_DATA_TOMBSTONED (TS fb0746b4d18f470f8815143973d6cf4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33736
I20240503 14:12:31.998872 5724 tablet_replica.cc:330] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: stopping tablet replica
I20240503 14:12:31.999081 5724 raft_consensus.cc:2229] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:31.999276 5724 raft_consensus.cc:2258] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.000128 5724 ts_tablet_manager.cc:1875] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:32.005396 5724 ts_tablet_manager.cc:1888] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:32.005564 5724 log.cc:1196] T 67f73e79bf0344bf9bbc2b1ecf3196a5 P fb0746b4d18f470f8815143973d6cf4d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/67f73e79bf0344bf9bbc2b1ecf3196a5
I20240503 14:12:32.006278 3117 catalog_manager.cc:4830] TS fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985): tablet 67f73e79bf0344bf9bbc2b1ecf3196a5 (table rebalance_test_table_1 [id=f2f3fdd1f0e04c8a8811861948f84daa]) successfully deleted
I20240503 14:12:32.018932 4518 consensus_queue.cc:237] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } }
I20240503 14:12:32.020092 5650 raft_consensus.cc:2932] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } } }
I20240503 14:12:32.023859 3116 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a78ff4930264840b7696ab966699f21 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:32.025357 3128 catalog_manager.cc:5484] T 5a78ff4930264840b7696ab966699f21 P 89194a0dee124ed1a36b1e81aba11e8c reported cstate change: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New cstate: current_term: 2 leader_uuid: "89194a0dee124ed1a36b1e81aba11e8c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "89194a0dee124ed1a36b1e81aba11e8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 34149 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:32.029162 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a78ff4930264840b7696ab966699f21 with delete_type TABLET_DATA_TOMBSTONED (TS f7f1ddc70d7f442296eb038048b1069d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46464
I20240503 14:12:32.029526 5751 tablet_replica.cc:330] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: stopping tablet replica
I20240503 14:12:32.029709 5751 raft_consensus.cc:2229] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:32.029979 5751 raft_consensus.cc:2258] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:32.030838 5751 ts_tablet_manager.cc:1875] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:32.036269 5751 ts_tablet_manager.cc:1888] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:32.036387 5751 log.cc:1196] T 5a78ff4930264840b7696ab966699f21 P f7f1ddc70d7f442296eb038048b1069d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/5a78ff4930264840b7696ab966699f21
I20240503 14:12:32.037238 3116 catalog_manager.cc:4830] TS f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751): tablet 5a78ff4930264840b7696ab966699f21 (table rebalance_test_table_0 [id=f76a7ef0c80846f88671272b90dc8175]) successfully deleted
I20240503 14:12:32.146965 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.153162 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.153700 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.175794 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.187605 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.215008 3252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:32.226122 3383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:32.405690 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.415060 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.419139 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.419521 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.455757 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:32.497272 3252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:32.498793 3383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:33.187603 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.211488 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.212286 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.216899 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.229943 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.232182 3252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:33.244887 3383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:33.467660 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.472510 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.475008 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.489786 3383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:33.498142 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.514853 3252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:33.515837 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.805636 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.819669 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.821820 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.828644 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.837822 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:33.840317 3252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:33.868018 3383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
f88c019087b3473da157be5d002064fa | 127.0.186.62:42623 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.186.20:38417 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
6c12721557f44ff28c4be1bcef9244c6 | 127.0.186.6:36477 | HEALTHY | <none> | 2 | 0
89194a0dee124ed1a36b1e81aba11e8c | 127.0.186.3:34149 | HEALTHY | <none> | 2 | 0
b10cbf74e3c34a6c8c1972c44e9d3d40 | 127.0.186.5:39315 | HEALTHY | <none> | 2 | 0
ed03e4970b2844df8fb33aa1a1b440c9 | 127.0.186.4:39891 | HEALTHY | <none> | 2 | 0
f42dc7cc6eec4c1e9f6b6f7721b5d2d9 | 127.0.186.7:43489 | HEALTHY | <none> | 2 | 0
f7f1ddc70d7f442296eb038048b1069d | 127.0.186.1:46751 | HEALTHY | <none> | 3 | 0
fb0746b4d18f470f8815143973d6cf4d | 127.0.186.2:41985 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.0.186.1 | experimental | 127.0.186.1:46751
local_ip_for_outbound_sockets | 127.0.186.2 | experimental | 127.0.186.2:41985
local_ip_for_outbound_sockets | 127.0.186.3 | experimental | 127.0.186.3:34149
local_ip_for_outbound_sockets | 127.0.186.4 | experimental | 127.0.186.4:39891
local_ip_for_outbound_sockets | 127.0.186.5 | experimental | 127.0.186.5:39315
local_ip_for_outbound_sockets | 127.0.186.6 | experimental | 127.0.186.6:36477
local_ip_for_outbound_sockets | 127.0.186.7 | experimental | 127.0.186.7:43489
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.186.1:46751
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.186.2:41985
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.186.3:34149
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.186.4:39891
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.186.5:39315
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.186.6:36477
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.186.7:43489
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.186.20:38417 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:12:33.938083 744 log_verifier.cc:126] Checking tablet 04bc99e3227e47779eeaa86a65a2f51c
I20240503 14:12:33.953464 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 04bc99e3227e47779eeaa86a65a2f51c
I20240503 14:12:33.953739 744 log_verifier.cc:126] Checking tablet 291abe2ff8214a3bac8cb9c98b8b20c2
I20240503 14:12:33.970046 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 291abe2ff8214a3bac8cb9c98b8b20c2
I20240503 14:12:33.970196 744 log_verifier.cc:126] Checking tablet 2ce9f735e2044c41900129f35549e7f4
I20240503 14:12:33.984501 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2ce9f735e2044c41900129f35549e7f4
I20240503 14:12:33.984656 744 log_verifier.cc:126] Checking tablet 551e225f487545e9bfbdde406910e735
I20240503 14:12:33.997437 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 551e225f487545e9bfbdde406910e735
I20240503 14:12:33.997582 744 log_verifier.cc:126] Checking tablet 5a78ff4930264840b7696ab966699f21
I20240503 14:12:34.011113 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5a78ff4930264840b7696ab966699f21
I20240503 14:12:34.011258 744 log_verifier.cc:126] Checking tablet 66432f3f144f40a48c8da0eacf2afe0f
I20240503 14:12:34.024183 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 66432f3f144f40a48c8da0eacf2afe0f
I20240503 14:12:34.024319 744 log_verifier.cc:126] Checking tablet 67f73e79bf0344bf9bbc2b1ecf3196a5
I20240503 14:12:34.037786 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 67f73e79bf0344bf9bbc2b1ecf3196a5
I20240503 14:12:34.037923 744 log_verifier.cc:126] Checking tablet 819948cd7188407a8a2210f01a2a4f7d
I20240503 14:12:34.049777 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 819948cd7188407a8a2210f01a2a4f7d
I20240503 14:12:34.049903 744 log_verifier.cc:126] Checking tablet 8acb48457026428da222d183814d769c
I20240503 14:12:34.062230 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8acb48457026428da222d183814d769c
I20240503 14:12:34.062361 744 log_verifier.cc:126] Checking tablet 8c8e2521b6e840ec880476c043408651
I20240503 14:12:34.074295 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8c8e2521b6e840ec880476c043408651
I20240503 14:12:34.074421 744 log_verifier.cc:126] Checking tablet 9dca255e05f4408e82caa70e2dc72af6
I20240503 14:12:34.087186 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9dca255e05f4408e82caa70e2dc72af6
I20240503 14:12:34.087318 744 log_verifier.cc:126] Checking tablet b6b48c2a49e249348b2a8c5109d251ce
I20240503 14:12:34.100519 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b6b48c2a49e249348b2a8c5109d251ce
I20240503 14:12:34.100661 744 log_verifier.cc:126] Checking tablet bdf123b6cdb64acfbdb132241ec11de0
I20240503 14:12:34.108263 744 log_verifier.cc:177] Verified matching terms for 2 ops in tablet bdf123b6cdb64acfbdb132241ec11de0
I20240503 14:12:34.108397 744 log_verifier.cc:126] Checking tablet c09ddc167734479ca650e195092f4eee
I20240503 14:12:34.120915 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c09ddc167734479ca650e195092f4eee
I20240503 14:12:34.121063 744 log_verifier.cc:126] Checking tablet c67b31603ba3418c99839bf4d9cc6eb0
I20240503 14:12:34.133427 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c67b31603ba3418c99839bf4d9cc6eb0
I20240503 14:12:34.133563 744 log_verifier.cc:126] Checking tablet d269db92e59e400db4db334fe177c4e6
I20240503 14:12:34.146606 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d269db92e59e400db4db334fe177c4e6
I20240503 14:12:34.149719 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.8:0
--local_ip_for_outbound_sockets=127.0.186.8
--webserver_interface=127.0.186.8
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--builtin_ntp_servers=127.0.186.20:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:34.505570 6038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:34.505887 6038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:34.505995 6038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:34.513833 6038 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:34.513986 6038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:34.514176 6038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.8
I20240503 14:12:34.523736 6038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.186.8
--webserver_port=0
--tserver_master_addrs=127.0.186.62:42623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.8
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:34.525107 6038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:34.526824 6038 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:34.540344 6044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:34.541321 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:34.542068 6043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:34.543375 6038 server_base.cc:1034] running on GCE node
I20240503 14:12:34.544291 6038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:34.545492 6038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:34.546774 6038 hybrid_clock.cc:648] HybridClock initialized: now 1714745554546717 us; error 48 us; skew 500 ppm
I20240503 14:12:34.547181 6038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:34.549885 6038 webserver.cc:460] Webserver started at http://127.0.186.8:43509/ using document root <none> and password file <none>
I20240503 14:12:34.550709 6038 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:34.550815 6038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:34.551160 6038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:34.554153 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "bf0d7cf302014d2a970f8b77b913dd29"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-mh2c"
I20240503 14:12:34.555034 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "bf0d7cf302014d2a970f8b77b913dd29"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-mh2c"
I20240503 14:12:34.561038 6038 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240503 14:12:34.565013 6052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.566582 6038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240503 14:12:34.566823 6038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "bf0d7cf302014d2a970f8b77b913dd29"
format_stamp: "Formatted at 2024-05-03 14:12:34 on dist-test-slave-mh2c"
I20240503 14:12:34.567052 6038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:34.588640 6038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:34.589543 6038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:34.589800 6038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:34.590780 6038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:34.592334 6038 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:34.592422 6038 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.592507 6038 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:34.592561 6038 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:34.625419 6038 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.8:36773
I20240503 14:12:34.625458 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.8:36773 every 8 connection(s)
I20240503 14:12:34.626796 6038 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:12:34.630703 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 6038
I20240503 14:12:34.630934 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:12:34.640234 6165 heartbeater.cc:347] Connected to a master server at 127.0.186.62:42623
I20240503 14:12:34.640568 6165 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:34.641418 6165 heartbeater.cc:511] Master 127.0.186.62:42623 requested a full tablet report, sending...
I20240503 14:12:34.643034 3131 ts_manager.cc:194] Registered new tserver with Master: bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8:36773)
I20240503 14:12:34.644265 3131 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.8:33937
I20240503 14:12:35.154831 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.156798 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.159720 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.175514 6100 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:35.209264 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.247984 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.268636 3252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:35.289039 3383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:35.545496 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.551672 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.566561 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.576042 6100 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:12:35.640915 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.643770 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:35.647292 6165 heartbeater.cc:503] Master 127.0.186.62:42623 was elected leader, sending a full tablet report...
I20240503 14:12:35.663262 3252 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:35.684840 3383 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:12:35.750824 3403 consensus_queue.cc:237] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } }
I20240503 14:12:35.753145 6258 raft_consensus.cc:2932] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } } }
I20240503 14:12:35.759279 3123 catalog_manager.cc:5484] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index -1 to 4, NON_VOTER bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:35.764432 3338 consensus_peers.cc:485] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d -> Peer bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8:36773): Couldn't send request to peer bf0d7cf302014d2a970f8b77b913dd29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c8e2521b6e840ec880476c043408651. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:35.776364 3272 consensus_queue.cc:237] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } }
I20240503 14:12:35.778360 6271 raft_consensus.cc:2932] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } } }
I20240503 14:12:35.783625 3123 catalog_manager.cc:5484] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d reported cstate change: config changed from index -1 to 4, NON_VOTER bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:35.786214 3207 consensus_peers.cc:485] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d -> Peer bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8:36773): Couldn't send request to peer bf0d7cf302014d2a970f8b77b913dd29. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 551e225f487545e9bfbdde406910e735. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:12:36.333791 6304 ts_tablet_manager.cc:902] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: Initiating tablet copy from peer f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:36.335821 6304 tablet_copy_client.cc:308] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:46751
I20240503 14:12:36.342924 3291 tablet_copy_service.cc:140] P f7f1ddc70d7f442296eb038048b1069d: Received BeginTabletCopySession request for tablet 551e225f487545e9bfbdde406910e735 from peer bf0d7cf302014d2a970f8b77b913dd29 ({username='slave'} at 127.0.186.8:44895)
I20240503 14:12:36.343151 3291 tablet_copy_service.cc:161] P f7f1ddc70d7f442296eb038048b1069d: Beginning new tablet copy session on tablet 551e225f487545e9bfbdde406910e735 from peer bf0d7cf302014d2a970f8b77b913dd29 at {username='slave'} at 127.0.186.8:44895: session id = bf0d7cf302014d2a970f8b77b913dd29-551e225f487545e9bfbdde406910e735
I20240503 14:12:36.344424 6311 ts_tablet_manager.cc:902] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: Initiating tablet copy from peer fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:36.346572 3291 tablet_copy_source_session.cc:205] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:36.348404 6311 tablet_copy_client.cc:308] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:41985
I20240503 14:12:36.352396 6304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551e225f487545e9bfbdde406910e735. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:36.356271 3422 tablet_copy_service.cc:140] P fb0746b4d18f470f8815143973d6cf4d: Received BeginTabletCopySession request for tablet 8c8e2521b6e840ec880476c043408651 from peer bf0d7cf302014d2a970f8b77b913dd29 ({username='slave'} at 127.0.186.8:37681)
I20240503 14:12:36.356485 3422 tablet_copy_service.cc:161] P fb0746b4d18f470f8815143973d6cf4d: Beginning new tablet copy session on tablet 8c8e2521b6e840ec880476c043408651 from peer bf0d7cf302014d2a970f8b77b913dd29 at {username='slave'} at 127.0.186.8:37681: session id = bf0d7cf302014d2a970f8b77b913dd29-8c8e2521b6e840ec880476c043408651
I20240503 14:12:36.359716 3422 tablet_copy_source_session.cc:205] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:12:36.361272 6311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c8e2521b6e840ec880476c043408651. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:36.366604 6304 tablet_copy_client.cc:783] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:36.366986 6304 tablet_copy_client.cc:647] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:36.367780 6311 tablet_copy_client.cc:783] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: tablet copy: Starting download of 0 data blocks...
I20240503 14:12:36.368001 6311 tablet_copy_client.cc:647] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: tablet copy: Starting download of 1 WAL segments...
I20240503 14:12:36.369504 6304 tablet_copy_client.cc:523] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:36.369966 6311 tablet_copy_client.cc:523] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:12:36.373734 6311 tablet_bootstrap.cc:492] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: Bootstrap starting.
I20240503 14:12:36.373757 6304 tablet_bootstrap.cc:492] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: Bootstrap starting.
I20240503 14:12:36.384460 6311 log.cc:824] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:36.418430 6311 tablet_bootstrap.cc:492] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:36.418978 6311 tablet_bootstrap.cc:492] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: Bootstrap complete.
I20240503 14:12:36.419433 6311 ts_tablet_manager.cc:1367] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: Time spent bootstrapping tablet: real 0.046s user 0.040s sys 0.004s
I20240503 14:12:36.422840 6304 tablet_bootstrap.cc:492] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:12:36.423213 6304 tablet_bootstrap.cc:492] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: Bootstrap complete.
I20240503 14:12:36.423463 6304 ts_tablet_manager.cc:1367] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: Time spent bootstrapping tablet: real 0.050s user 0.036s sys 0.008s
I20240503 14:12:36.424355 6304 raft_consensus.cc:348] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } }
I20240503 14:12:36.424360 6311 raft_consensus.cc:348] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } }
I20240503 14:12:36.424772 6311 raft_consensus.cc:729] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf0d7cf302014d2a970f8b77b913dd29, State: Initialized, Role: LEARNER
I20240503 14:12:36.424777 6304 raft_consensus.cc:729] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bf0d7cf302014d2a970f8b77b913dd29, State: Initialized, Role: LEARNER
I20240503 14:12:36.425266 6304 consensus_queue.cc:260] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } }
I20240503 14:12:36.425272 6311 consensus_queue.cc:260] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: true } }
I20240503 14:12:36.426806 6311 ts_tablet_manager.cc:1398] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29: Time spent starting tablet: real 0.007s user 0.010s sys 0.000s
I20240503 14:12:36.426923 6304 ts_tablet_manager.cc:1398] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29: Time spent starting tablet: real 0.003s user 0.002s sys 0.004s
I20240503 14:12:36.427796 3422 tablet_copy_service.cc:342] P fb0746b4d18f470f8815143973d6cf4d: Request end of tablet copy session bf0d7cf302014d2a970f8b77b913dd29-8c8e2521b6e840ec880476c043408651 received from {username='slave'} at 127.0.186.8:37681
I20240503 14:12:36.427961 3422 tablet_copy_service.cc:433] P fb0746b4d18f470f8815143973d6cf4d: ending tablet copy session bf0d7cf302014d2a970f8b77b913dd29-8c8e2521b6e840ec880476c043408651 on tablet 8c8e2521b6e840ec880476c043408651 with peer bf0d7cf302014d2a970f8b77b913dd29
I20240503 14:12:36.427986 3291 tablet_copy_service.cc:342] P f7f1ddc70d7f442296eb038048b1069d: Request end of tablet copy session bf0d7cf302014d2a970f8b77b913dd29-551e225f487545e9bfbdde406910e735 received from {username='slave'} at 127.0.186.8:44895
I20240503 14:12:36.428117 3291 tablet_copy_service.cc:433] P f7f1ddc70d7f442296eb038048b1069d: ending tablet copy session bf0d7cf302014d2a970f8b77b913dd29-551e225f487545e9bfbdde406910e735 on tablet 551e225f487545e9bfbdde406910e735 with peer bf0d7cf302014d2a970f8b77b913dd29
I20240503 14:12:36.739807 6120 raft_consensus.cc:1203] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:36.940935 6120 raft_consensus.cc:1203] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:12:37.300158 6365 raft_consensus.cc:1050] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: attempting to promote NON_VOTER bf0d7cf302014d2a970f8b77b913dd29 to VOTER
I20240503 14:12:37.300711 6365 consensus_queue.cc:237] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:37.302549 6120 raft_consensus.cc:1261] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 LEARNER]: Refusing update from remote peer fb0746b4d18f470f8815143973d6cf4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:37.303184 6365 consensus_queue.cc:1035] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:37.309345 6366 raft_consensus.cc:2932] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } } }
I20240503 14:12:37.310379 6120 raft_consensus.cc:2932] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } } }
I20240503 14:12:37.314321 3124 catalog_manager.cc:5484] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d reported cstate change: config changed from index 4 to 5, bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fb0746b4d18f470f8815143973d6cf4d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:37.350019 6354 raft_consensus.cc:1050] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: attempting to promote NON_VOTER bf0d7cf302014d2a970f8b77b913dd29 to VOTER
I20240503 14:12:37.350761 6354 consensus_queue.cc:237] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:37.352931 6120 raft_consensus.cc:1261] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 LEARNER]: Refusing update from remote peer f7f1ddc70d7f442296eb038048b1069d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:12:37.353626 6353 consensus_queue.cc:1035] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:12:37.356889 6354 raft_consensus.cc:2932] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } } }
I20240503 14:12:37.360841 6120 raft_consensus.cc:2932] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } } }
I20240503 14:12:37.365603 3123 catalog_manager.cc:5484] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 reported cstate change: config changed from index 4 to 5, bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f7f1ddc70d7f442296eb038048b1069d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } } }
I20240503 14:12:37.379951 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c8e2521b6e840ec880476c043408651"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58316
I20240503 14:12:37.380231 3403 raft_consensus.cc:595] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:37.599973 3272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551e225f487545e9bfbdde406910e735"
dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39638
I20240503 14:12:37.600181 3272 raft_consensus.cc:595] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:37.617962 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c8e2521b6e840ec880476c043408651"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58338
I20240503 14:12:37.618191 3403 raft_consensus.cc:595] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:37.844278 3272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "551e225f487545e9bfbdde406910e735"
dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39650
I20240503 14:12:37.844565 3272 raft_consensus.cc:595] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:37.863127 3403 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c8e2521b6e840ec880476c043408651"
dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58362
I20240503 14:12:37.863328 3403 raft_consensus.cc:595] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Received request to transfer leadership
I20240503 14:12:37.933159 6432 raft_consensus.cc:979] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: : Instructing follower bf0d7cf302014d2a970f8b77b913dd29 to start an election
I20240503 14:12:37.933326 6432 raft_consensus.cc:1067] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Signalling peer bf0d7cf302014d2a970f8b77b913dd29 to start an election
I20240503 14:12:37.934120 6120 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c8e2521b6e840ec880476c043408651"
dest_uuid: "bf0d7cf302014d2a970f8b77b913dd29"
from {username='slave'} at 127.0.186.2:43807
I20240503 14:12:37.934386 6120 raft_consensus.cc:482] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:37.934532 6120 raft_consensus.cc:3037] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:37.936512 6120 raft_consensus.cc:504] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:37.937726 6120 leader_election.cc:288] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [CANDIDATE]: Term 2 election: Requested vote from peers fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985)
I20240503 14:12:37.938282 3403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c8e2521b6e840ec880476c043408651" candidate_uuid: "bf0d7cf302014d2a970f8b77b913dd29" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb0746b4d18f470f8815143973d6cf4d"
I20240503 14:12:37.938470 3403 raft_consensus.cc:3032] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:37.938534 3403 raft_consensus.cc:729] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fb0746b4d18f470f8815143973d6cf4d, State: Running, Role: LEADER
I20240503 14:12:37.938767 3403 consensus_queue.cc:260] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:37.938994 3403 raft_consensus.cc:3037] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:37.941074 3403 raft_consensus.cc:2454] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0d7cf302014d2a970f8b77b913dd29 in term 2.
I20240503 14:12:37.941628 6055 leader_election.cc:302] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: bf0d7cf302014d2a970f8b77b913dd29, fb0746b4d18f470f8815143973d6cf4d; no voters:
I20240503 14:12:37.942291 6433 raft_consensus.cc:2781] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:37.942653 6433 raft_consensus.cc:686] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 2 LEADER]: Becoming Leader. State: Replica: bf0d7cf302014d2a970f8b77b913dd29, State: Running, Role: LEADER
I20240503 14:12:37.943133 6433 consensus_queue.cc:237] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:37.946134 3124 catalog_manager.cc:5484] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 reported cstate change: term changed from 1 to 2, leader changed from fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) to bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8). New cstate: current_term: 2 leader_uuid: "bf0d7cf302014d2a970f8b77b913dd29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:12:37.956197 3117 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 8c8e2521b6e840ec880476c043408651 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8:36773) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 14:12:37.981958 6441 raft_consensus.cc:979] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: : Instructing follower bf0d7cf302014d2a970f8b77b913dd29 to start an election
I20240503 14:12:37.982136 6441 raft_consensus.cc:1067] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Signalling peer bf0d7cf302014d2a970f8b77b913dd29 to start an election
I20240503 14:12:37.982848 6120 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "551e225f487545e9bfbdde406910e735"
dest_uuid: "bf0d7cf302014d2a970f8b77b913dd29"
from {username='slave'} at 127.0.186.1:37069
I20240503 14:12:37.983103 6120 raft_consensus.cc:482] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:12:37.983208 6120 raft_consensus.cc:3037] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:37.985215 6120 raft_consensus.cc:504] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:37.985801 6120 leader_election.cc:288] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [CANDIDATE]: Term 2 election: Requested vote from peers f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751)
I20240503 14:12:37.986449 3272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551e225f487545e9bfbdde406910e735" candidate_uuid: "bf0d7cf302014d2a970f8b77b913dd29" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f7f1ddc70d7f442296eb038048b1069d"
I20240503 14:12:37.986690 3272 raft_consensus.cc:3032] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:12:37.986824 3272 raft_consensus.cc:729] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f7f1ddc70d7f442296eb038048b1069d, State: Running, Role: LEADER
I20240503 14:12:37.987098 3272 consensus_queue.cc:260] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:37.987351 3272 raft_consensus.cc:3037] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:12:37.989557 3272 raft_consensus.cc:2454] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf0d7cf302014d2a970f8b77b913dd29 in term 2.
I20240503 14:12:37.990094 6054 leader_election.cc:302] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: bf0d7cf302014d2a970f8b77b913dd29, f7f1ddc70d7f442296eb038048b1069d; no voters:
I20240503 14:12:37.990399 6433 raft_consensus.cc:2781] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:12:37.990572 6433 raft_consensus.cc:686] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 2 LEADER]: Becoming Leader. State: Replica: bf0d7cf302014d2a970f8b77b913dd29, State: Running, Role: LEADER
I20240503 14:12:37.990852 6433 consensus_queue.cc:237] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:37.992848 3124 catalog_manager.cc:5484] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 reported cstate change: term changed from 1 to 2, leader changed from f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) to bf0d7cf302014d2a970f8b77b913dd29 (127.0.186.8). New cstate: current_term: 2 leader_uuid: "bf0d7cf302014d2a970f8b77b913dd29" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:38.507262 3272 raft_consensus.cc:1261] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Refusing update from remote peer bf0d7cf302014d2a970f8b77b913dd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:38.508123 6462 consensus_queue.cc:1035] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7f1ddc70d7f442296eb038048b1069d" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 46751 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:38.519872 3403 raft_consensus.cc:1261] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Refusing update from remote peer bf0d7cf302014d2a970f8b77b913dd29: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:12:38.520532 6473 consensus_queue.cc:1035] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb0746b4d18f470f8815143973d6cf4d" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 41985 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:12:38.585582 6120 consensus_queue.cc:237] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:38.586639 6462 raft_consensus.cc:2932] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } } }
I20240503 14:12:38.589390 3117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c8e2521b6e840ec880476c043408651 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:38.590937 3123 catalog_manager.cc:5484] T 8c8e2521b6e840ec880476c043408651 P bf0d7cf302014d2a970f8b77b913dd29 reported cstate change: config changed from index 5 to 7, VOTER fb0746b4d18f470f8815143973d6cf4d (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "bf0d7cf302014d2a970f8b77b913dd29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:38.595093 3383 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c8e2521b6e840ec880476c043408651 with delete_type TABLET_DATA_TOMBSTONED (TS fb0746b4d18f470f8815143973d6cf4d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33736
I20240503 14:12:38.595846 6485 tablet_replica.cc:330] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: stopping tablet replica
I20240503 14:12:38.596037 6485 raft_consensus.cc:2229] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:38.596200 6485 raft_consensus.cc:2258] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:38.597656 6485 ts_tablet_manager.cc:1875] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:38.601783 6485 ts_tablet_manager.cc:1888] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:38.601886 6485 log.cc:1196] T 8c8e2521b6e840ec880476c043408651 P fb0746b4d18f470f8815143973d6cf4d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/8c8e2521b6e840ec880476c043408651
I20240503 14:12:38.602635 3117 catalog_manager.cc:4830] TS fb0746b4d18f470f8815143973d6cf4d (127.0.186.2:41985): tablet 8c8e2521b6e840ec880476c043408651 (table rebalance_test_table_3 [id=a7ffaa5ef2a24eabac1144221dcd708c]) successfully deleted
I20240503 14:12:38.626439 6120 consensus_queue.cc:237] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } }
I20240503 14:12:38.627408 6473 raft_consensus.cc:2932] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } } }
I20240503 14:12:38.630283 3117 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 551e225f487545e9bfbdde406910e735 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 14:12:38.631613 3123 catalog_manager.cc:5484] T 551e225f487545e9bfbdde406910e735 P bf0d7cf302014d2a970f8b77b913dd29 reported cstate change: config changed from index 5 to 7, VOTER f7f1ddc70d7f442296eb038048b1069d (127.0.186.1) evicted. New cstate: current_term: 2 leader_uuid: "bf0d7cf302014d2a970f8b77b913dd29" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bf0d7cf302014d2a970f8b77b913dd29" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36773 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:12:38.635001 3252 tablet_service.cc:1514] Processing DeleteTablet for tablet 551e225f487545e9bfbdde406910e735 with delete_type TABLET_DATA_TOMBSTONED (TS f7f1ddc70d7f442296eb038048b1069d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:46464
I20240503 14:12:38.635689 6486 tablet_replica.cc:330] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: stopping tablet replica
I20240503 14:12:38.635876 6486 raft_consensus.cc:2229] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:12:38.636054 6486 raft_consensus.cc:2258] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:12:38.637519 6486 ts_tablet_manager.cc:1875] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:12:38.641656 6486 ts_tablet_manager.cc:1888] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:12:38.641778 6486 log.cc:1196] T 551e225f487545e9bfbdde406910e735 P f7f1ddc70d7f442296eb038048b1069d: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/551e225f487545e9bfbdde406910e735
I20240503 14:12:38.642545 3116 catalog_manager.cc:4830] TS f7f1ddc70d7f442296eb038048b1069d (127.0.186.1:46751): tablet 551e225f487545e9bfbdde406910e735 (table rebalance_test_table_4 [id=50e14958ddc64d79a9d091ec5d86e98c]) successfully deleted
I20240503 14:12:38.873795 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.874675 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.876061 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.886888 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.888419 6100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.899911 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.903209 3383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:38.915416 3252 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.101716 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.106492 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.113978 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.123849 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.123979 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.124574 3252 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.168581 6100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.187336 3383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.468016 4497 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.471312 4898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.478956 4768 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.481376 6100 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.484299 4638 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.522701 5028 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.541617 3252 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:12:39.549840 3383 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
f88c019087b3473da157be5d002064fa | 127.0.186.62:42623 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.186.20:38417 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
6c12721557f44ff28c4be1bcef9244c6 | 127.0.186.6:36477 | HEALTHY | <none> | 2 | 0
89194a0dee124ed1a36b1e81aba11e8c | 127.0.186.3:34149 | HEALTHY | <none> | 2 | 0
b10cbf74e3c34a6c8c1972c44e9d3d40 | 127.0.186.5:39315 | HEALTHY | <none> | 2 | 0
bf0d7cf302014d2a970f8b77b913dd29 | 127.0.186.8:36773 | HEALTHY | <none> | 2 | 0
ed03e4970b2844df8fb33aa1a1b440c9 | 127.0.186.4:39891 | HEALTHY | <none> | 2 | 0
f42dc7cc6eec4c1e9f6b6f7721b5d2d9 | 127.0.186.7:43489 | HEALTHY | <none> | 2 | 0
f7f1ddc70d7f442296eb038048b1069d | 127.0.186.1:46751 | HEALTHY | <none> | 2 | 0
fb0746b4d18f470f8815143973d6cf4d | 127.0.186.2:41985 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.0.186.1 | experimental | 127.0.186.1:46751
local_ip_for_outbound_sockets | 127.0.186.2 | experimental | 127.0.186.2:41985
local_ip_for_outbound_sockets | 127.0.186.3 | experimental | 127.0.186.3:34149
local_ip_for_outbound_sockets | 127.0.186.4 | experimental | 127.0.186.4:39891
local_ip_for_outbound_sockets | 127.0.186.5 | experimental | 127.0.186.5:39315
local_ip_for_outbound_sockets | 127.0.186.6 | experimental | 127.0.186.6:36477
local_ip_for_outbound_sockets | 127.0.186.7 | experimental | 127.0.186.7:43489
local_ip_for_outbound_sockets | 127.0.186.8 | experimental | 127.0.186.8:36773
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.186.1:46751
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.186.2:41985
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.186.3:34149
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.186.4:39891
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.186.5:39315
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.0.186.6:36477
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.0.186.7:43489
server_dump_info_path | /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.0.186.8:36773
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.186.20:38417 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 14:12:39.607365 744 log_verifier.cc:126] Checking tablet 04bc99e3227e47779eeaa86a65a2f51c
I20240503 14:12:39.617425 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 04bc99e3227e47779eeaa86a65a2f51c
I20240503 14:12:39.617527 744 log_verifier.cc:126] Checking tablet 291abe2ff8214a3bac8cb9c98b8b20c2
I20240503 14:12:39.627681 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 291abe2ff8214a3bac8cb9c98b8b20c2
I20240503 14:12:39.627782 744 log_verifier.cc:126] Checking tablet 2ce9f735e2044c41900129f35549e7f4
I20240503 14:12:39.639230 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 2ce9f735e2044c41900129f35549e7f4
I20240503 14:12:39.639346 744 log_verifier.cc:126] Checking tablet 551e225f487545e9bfbdde406910e735
I20240503 14:12:39.652005 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 551e225f487545e9bfbdde406910e735
I20240503 14:12:39.652112 744 log_verifier.cc:126] Checking tablet 5a78ff4930264840b7696ab966699f21
I20240503 14:12:39.662075 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5a78ff4930264840b7696ab966699f21
I20240503 14:12:39.662174 744 log_verifier.cc:126] Checking tablet 66432f3f144f40a48c8da0eacf2afe0f
I20240503 14:12:39.671945 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 66432f3f144f40a48c8da0eacf2afe0f
I20240503 14:12:39.672118 744 log_verifier.cc:126] Checking tablet 67f73e79bf0344bf9bbc2b1ecf3196a5
I20240503 14:12:39.682919 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 67f73e79bf0344bf9bbc2b1ecf3196a5
I20240503 14:12:39.683015 744 log_verifier.cc:126] Checking tablet 819948cd7188407a8a2210f01a2a4f7d
I20240503 14:12:39.692518 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 819948cd7188407a8a2210f01a2a4f7d
I20240503 14:12:39.692627 744 log_verifier.cc:126] Checking tablet 8acb48457026428da222d183814d769c
I20240503 14:12:39.707634 744 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8acb48457026428da222d183814d769c
I20240503 14:12:39.707754 744 log_verifier.cc:126] Checking tablet 8c8e2521b6e840ec880476c043408651
I20240503 14:12:39.722352 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 8c8e2521b6e840ec880476c043408651
I20240503 14:12:39.722458 744 log_verifier.cc:126] Checking tablet 9dca255e05f4408e82caa70e2dc72af6
I20240503 14:12:39.732532 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9dca255e05f4408e82caa70e2dc72af6
I20240503 14:12:39.732635 744 log_verifier.cc:126] Checking tablet b6b48c2a49e249348b2a8c5109d251ce
I20240503 14:12:39.742923 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet b6b48c2a49e249348b2a8c5109d251ce
I20240503 14:12:39.743028 744 log_verifier.cc:126] Checking tablet bdf123b6cdb64acfbdb132241ec11de0
I20240503 14:12:39.748220 744 log_verifier.cc:177] Verified matching terms for 2 ops in tablet bdf123b6cdb64acfbdb132241ec11de0
I20240503 14:12:39.748308 744 log_verifier.cc:126] Checking tablet c09ddc167734479ca650e195092f4eee
I20240503 14:12:39.761758 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c09ddc167734479ca650e195092f4eee
I20240503 14:12:39.761874 744 log_verifier.cc:126] Checking tablet c67b31603ba3418c99839bf4d9cc6eb0
I20240503 14:12:39.775501 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c67b31603ba3418c99839bf4d9cc6eb0
I20240503 14:12:39.775651 744 log_verifier.cc:126] Checking tablet d269db92e59e400db4db334fe177c4e6
I20240503 14:12:39.786139 744 log_verifier.cc:177] Verified matching terms for 7 ops in tablet d269db92e59e400db4db334fe177c4e6
I20240503 14:12:39.787021 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 3190
I20240503 14:12:39.788519 3312 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:39.929159 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 3190
I20240503 14:12:39.954576 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 3321
I20240503 14:12:39.955657 3442 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:40.096124 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 3321
I20240503 14:12:40.120479 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 4434
I20240503 14:12:40.127063 4557 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:40.241657 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 4434
I20240503 14:12:40.262557 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 4570
I20240503 14:12:40.268909 4697 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:40.394223 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 4570
I20240503 14:12:40.415374 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 4706
I20240503 14:12:40.422161 4827 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:40.540722 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 4706
I20240503 14:12:40.561913 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 4836
I20240503 14:12:40.569093 4958 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:40.685511 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 4836
I20240503 14:12:40.705955 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 4966
I20240503 14:12:40.712036 5088 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:40.827383 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 4966
I20240503 14:12:40.848160 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 6038
I20240503 14:12:40.854584 6160 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:40.956459 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 6038
I20240503 14:12:40.975747 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 3100
I20240503 14:12:40.976898 3161 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:41.129155 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 3100
2024-05-03T14:12:41Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (28339 ms)
[----------] 1 test from RebalanceParamTest (28339 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240503 14:12:41.198818 744 test_util.cc:268] Using random seed: 1343107273
I20240503 14:12:41.200773 744 ts_itest-base.cc:112] Starting cluster with:
I20240503 14:12:41.200845 744 ts_itest-base.cc:113] --------------
I20240503 14:12:41.200907 744 ts_itest-base.cc:114] 8 tablet servers
I20240503 14:12:41.200953 744 ts_itest-base.cc:115] 3 replicas per TS
I20240503 14:12:41.201020 744 ts_itest-base.cc:116] --------------
2024-05-03T14:12:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T14:12:41Z Disabled control of system clock
I20240503 14:12:41.231711 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.186.62:45803
--webserver_interface=127.0.186.62
--webserver_port=0
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.186.62:45803
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 14:12:41.571496 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:41.571877 6628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:41.572000 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:41.579627 6628 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:41.579774 6628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 14:12:41.579850 6628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:41.579938 6628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 14:12:41.580004 6628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 14:12:41.589377 6628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.0.186.62:45803
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.186.62:45803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.186.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:41.590826 6628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:41.592751 6628 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:41.604573 6633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:41.605242 6634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:41.606171 6636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:41.607055 6628 server_base.cc:1034] running on GCE node
I20240503 14:12:41.607952 6628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:41.609478 6628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:41.610705 6628 hybrid_clock.cc:648] HybridClock initialized: now 1714745561610654 us; error 47 us; skew 500 ppm
I20240503 14:12:41.611075 6628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:41.613528 6628 webserver.cc:460] Webserver started at http://127.0.186.62:44457/ using document root <none> and password file <none>
I20240503 14:12:41.614329 6628 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:41.614449 6628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:41.614777 6628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:41.617516 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "371d064947984511bcd526e0de3fb910"
format_stamp: "Formatted at 2024-05-03 14:12:41 on dist-test-slave-mh2c"
I20240503 14:12:41.618305 6628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "371d064947984511bcd526e0de3fb910"
format_stamp: "Formatted at 2024-05-03 14:12:41 on dist-test-slave-mh2c"
I20240503 14:12:41.623929 6628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 14:12:41.627549 6642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:41.629181 6628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:41.629371 6628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
uuid: "371d064947984511bcd526e0de3fb910"
format_stamp: "Formatted at 2024-05-03 14:12:41 on dist-test-slave-mh2c"
I20240503 14:12:41.629580 6628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:41.656971 6628 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:41.657896 6628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:41.658201 6628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:41.676725 6628 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.62:45803
I20240503 14:12:41.676743 6693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.62:45803 every 8 connection(s)
I20240503 14:12:41.678361 6628 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 14:12:41.682129 6694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:41.682958 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 6628
I20240503 14:12:41.683156 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 14:12:41.693127 6694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Bootstrap starting.
I20240503 14:12:41.696643 6694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:41.697696 6694 log.cc:824] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:41.700264 6694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: No bootstrap required, opened a new log
I20240503 14:12:41.706545 6694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371d064947984511bcd526e0de3fb910" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45803 } }
I20240503 14:12:41.706925 6694 raft_consensus.cc:374] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:41.707034 6694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 371d064947984511bcd526e0de3fb910, State: Initialized, Role: FOLLOWER
I20240503 14:12:41.707696 6694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371d064947984511bcd526e0de3fb910" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45803 } }
I20240503 14:12:41.707885 6694 raft_consensus.cc:388] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 14:12:41.708019 6694 raft_consensus.cc:482] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 14:12:41.708168 6694 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:41.710462 6694 raft_consensus.cc:504] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371d064947984511bcd526e0de3fb910" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45803 } }
I20240503 14:12:41.710942 6694 leader_election.cc:302] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 371d064947984511bcd526e0de3fb910; no voters:
I20240503 14:12:41.711555 6694 leader_election.cc:288] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 14:12:41.711733 6699 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:41.712353 6699 raft_consensus.cc:686] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [term 1 LEADER]: Becoming Leader. State: Replica: 371d064947984511bcd526e0de3fb910, State: Running, Role: LEADER
I20240503 14:12:41.712898 6699 consensus_queue.cc:237] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371d064947984511bcd526e0de3fb910" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45803 } }
I20240503 14:12:41.713595 6694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 14:12:41.715862 6701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 371d064947984511bcd526e0de3fb910. Latest consensus state: current_term: 1 leader_uuid: "371d064947984511bcd526e0de3fb910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371d064947984511bcd526e0de3fb910" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45803 } } }
I20240503 14:12:41.715951 6700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "371d064947984511bcd526e0de3fb910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "371d064947984511bcd526e0de3fb910" member_type: VOTER last_known_addr { host: "127.0.186.62" port: 45803 } } }
I20240503 14:12:41.716235 6701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:41.716310 6700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910 [sys.catalog]: This master's current role is: LEADER
I20240503 14:12:41.718425 6704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 14:12:41.723917 6704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 14:12:41.732848 6704 catalog_manager.cc:1347] Generated new cluster ID: b46be8c2585b4492b0059acb575d7bec
I20240503 14:12:41.732982 6704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 14:12:41.761994 6704 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 14:12:41.763311 6704 catalog_manager.cc:1504] Loading token signing keys...
I20240503 14:12:41.780732 6704 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Generated new TSK 0
I20240503 14:12:41.781622 6704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 14:12:41.789350 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.1:0
--local_ip_for_outbound_sockets=127.0.186.1
--webserver_interface=127.0.186.1
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:12:42.153959 6718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:42.154296 6718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:42.154469 6718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:42.162174 6718 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:42.162307 6718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:42.162473 6718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.1
I20240503 14:12:42.171783 6718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.186.1
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.1
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:42.173261 6718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:42.175161 6718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 14:12:42.189291 6718 server_base.cc:1034] running on GCE node
W20240503 14:12:42.189657 6723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:42.190522 6726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:42.190663 6724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:42.191412 6718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:42.192432 6718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:42.193641 6718 hybrid_clock.cc:648] HybridClock initialized: now 1714745562193594 us; error 40 us; skew 500 ppm
I20240503 14:12:42.194000 6718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:42.200677 6718 webserver.cc:460] Webserver started at http://127.0.186.1:45529/ using document root <none> and password file <none>
I20240503 14:12:42.201465 6718 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:42.201589 6718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:42.201918 6718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:42.204618 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f1a669e28f014fe287932acf56bebc13"
format_stamp: "Formatted at 2024-05-03 14:12:42 on dist-test-slave-mh2c"
I20240503 14:12:42.205446 6718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f1a669e28f014fe287932acf56bebc13"
format_stamp: "Formatted at 2024-05-03 14:12:42 on dist-test-slave-mh2c"
I20240503 14:12:42.210657 6718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 14:12:42.214524 6732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:42.216163 6718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:42.216336 6718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f1a669e28f014fe287932acf56bebc13"
format_stamp: "Formatted at 2024-05-03 14:12:42 on dist-test-slave-mh2c"
I20240503 14:12:42.216543 6718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:42.231309 6718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:42.232292 6718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:42.232585 6718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:42.233614 6718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:42.235280 6718 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:42.235383 6718 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:42.235515 6718 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:42.235623 6718 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:42.269505 6718 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.1:34101
I20240503 14:12:42.269562 6844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.1:34101 every 8 connection(s)
I20240503 14:12:42.270884 6718 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 14:12:42.274024 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 6718
I20240503 14:12:42.274248 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 14:12:42.277707 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.2:0
--local_ip_for_outbound_sockets=127.0.186.2
--webserver_interface=127.0.186.2
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:42.285737 6845 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:12:42.286109 6845 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:42.287264 6845 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:12:42.289718 6659 ts_manager.cc:194] Registered new tserver with Master: f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:12:42.291321 6659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.1:43629
W20240503 14:12:42.620975 6849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:42.621330 6849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:42.621479 6849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:42.629228 6849 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:42.629357 6849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:42.629521 6849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.2
I20240503 14:12:42.638799 6849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.186.2
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.2
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:42.640237 6849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:42.642027 6849 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:42.654981 6854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:42.655704 6857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:42.656301 6855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:42.657303 6849 server_base.cc:1034] running on GCE node
I20240503 14:12:42.657971 6849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:42.658987 6849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:42.660198 6849 hybrid_clock.cc:648] HybridClock initialized: now 1714745562660139 us; error 50 us; skew 500 ppm
I20240503 14:12:42.660532 6849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:42.662917 6849 webserver.cc:460] Webserver started at http://127.0.186.2:35567/ using document root <none> and password file <none>
I20240503 14:12:42.663710 6849 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:42.663828 6849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:42.664140 6849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:42.666731 6849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7186ed8510424fd597173eaf10c57b12"
format_stamp: "Formatted at 2024-05-03 14:12:42 on dist-test-slave-mh2c"
I20240503 14:12:42.667495 6849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7186ed8510424fd597173eaf10c57b12"
format_stamp: "Formatted at 2024-05-03 14:12:42 on dist-test-slave-mh2c"
I20240503 14:12:42.673267 6849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20240503 14:12:42.677173 6863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:42.678911 6849 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:12:42.679100 6849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7186ed8510424fd597173eaf10c57b12"
format_stamp: "Formatted at 2024-05-03 14:12:42 on dist-test-slave-mh2c"
I20240503 14:12:42.679323 6849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:42.705579 6849 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:42.706429 6849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:42.706673 6849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:42.707682 6849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:42.709209 6849 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:42.709309 6849 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:42.709436 6849 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:42.709510 6849 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:42.742038 6849 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.2:45285
I20240503 14:12:42.742097 6975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.2:45285 every 8 connection(s)
I20240503 14:12:42.743358 6849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 14:12:42.751669 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 6849
I20240503 14:12:42.751843 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 14:12:42.755098 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.3:0
--local_ip_for_outbound_sockets=127.0.186.3
--webserver_interface=127.0.186.3
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:42.756444 6976 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:12:42.756758 6976 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:42.757601 6976 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:12:42.759171 6659 ts_manager.cc:194] Registered new tserver with Master: 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:42.760303 6659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.2:39969
W20240503 14:12:43.094113 6979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:43.094470 6979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:43.094605 6979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:43.102375 6979 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:43.102506 6979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:43.102679 6979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.3
I20240503 14:12:43.112043 6979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.186.3
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.3
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:43.113437 6979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:43.115283 6979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:43.128659 6984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:43.129287 6987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:43.130188 6985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:43.130898 6979 server_base.cc:1034] running on GCE node
I20240503 14:12:43.131547 6979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:43.132581 6979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:43.133796 6979 hybrid_clock.cc:648] HybridClock initialized: now 1714745563133755 us; error 42 us; skew 500 ppm
I20240503 14:12:43.134120 6979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:43.136545 6979 webserver.cc:460] Webserver started at http://127.0.186.3:32911/ using document root <none> and password file <none>
I20240503 14:12:43.137326 6979 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:43.137445 6979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:43.137758 6979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:43.142270 6979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "09723f236d8044538d920a37b5d9aa8c"
format_stamp: "Formatted at 2024-05-03 14:12:43 on dist-test-slave-mh2c"
I20240503 14:12:43.143060 6979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "09723f236d8044538d920a37b5d9aa8c"
format_stamp: "Formatted at 2024-05-03 14:12:43 on dist-test-slave-mh2c"
I20240503 14:12:43.148387 6979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240503 14:12:43.151991 6993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:43.153462 6979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:12:43.153632 6979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "09723f236d8044538d920a37b5d9aa8c"
format_stamp: "Formatted at 2024-05-03 14:12:43 on dist-test-slave-mh2c"
I20240503 14:12:43.153839 6979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:43.187463 6979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:43.188416 6979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:43.188683 6979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:43.189661 6979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:43.191192 6979 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:43.191290 6979 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:43.191412 6979 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:43.191485 6979 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:43.224938 6979 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.3:44691
I20240503 14:12:43.225008 7105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.3:44691 every 8 connection(s)
I20240503 14:12:43.226326 6979 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 14:12:43.228107 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 6979
I20240503 14:12:43.228286 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 14:12:43.231324 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.4:0
--local_ip_for_outbound_sockets=127.0.186.4
--webserver_interface=127.0.186.4
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:43.240919 7106 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:12:43.241259 7106 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:43.242146 7106 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:12:43.243726 6659 ts_manager.cc:194] Registered new tserver with Master: 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:43.244561 6659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.3:36837
I20240503 14:12:43.295238 6845 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
W20240503 14:12:43.575404 7109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:43.575855 7109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:43.576022 7109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:43.584676 7109 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:43.584808 7109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:43.584970 7109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.4
I20240503 14:12:43.594843 7109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.186.4
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.4
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:43.596307 7109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:43.598052 7109 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:43.611105 7114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:43.611980 7117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:43.612882 7115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:43.613775 7109 server_base.cc:1034] running on GCE node
I20240503 14:12:43.614478 7109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:43.615501 7109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:43.616720 7109 hybrid_clock.cc:648] HybridClock initialized: now 1714745563616664 us; error 51 us; skew 500 ppm
I20240503 14:12:43.617069 7109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:43.619374 7109 webserver.cc:460] Webserver started at http://127.0.186.4:34573/ using document root <none> and password file <none>
I20240503 14:12:43.620189 7109 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:43.620307 7109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:43.620607 7109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:43.623329 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e1bf5fb7402c46829974e2f74cd88708"
format_stamp: "Formatted at 2024-05-03 14:12:43 on dist-test-slave-mh2c"
I20240503 14:12:43.624171 7109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e1bf5fb7402c46829974e2f74cd88708"
format_stamp: "Formatted at 2024-05-03 14:12:43 on dist-test-slave-mh2c"
I20240503 14:12:43.629432 7109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:12:43.633038 7123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:43.634546 7109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 14:12:43.634716 7109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e1bf5fb7402c46829974e2f74cd88708"
format_stamp: "Formatted at 2024-05-03 14:12:43 on dist-test-slave-mh2c"
I20240503 14:12:43.634927 7109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:43.655372 7109 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:43.656283 7109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:43.656546 7109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:43.657495 7109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:43.659065 7109 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:43.659148 7109 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:43.659265 7109 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:43.659334 7109 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:43.692579 7109 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.4:44337
I20240503 14:12:43.692642 7235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.4:44337 every 8 connection(s)
I20240503 14:12:43.693931 7109 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 14:12:43.694535 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 7109
I20240503 14:12:43.694693 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 14:12:43.698068 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.5:0
--local_ip_for_outbound_sockets=127.0.186.5
--webserver_interface=127.0.186.5
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:43.708774 7236 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:12:43.709139 7236 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:43.710232 7236 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:12:43.711894 6659 ts_manager.cc:194] Registered new tserver with Master: e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:43.712800 6659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.4:36717
I20240503 14:12:43.763677 6976 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
W20240503 14:12:44.043452 7239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:44.043820 7239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:44.043985 7239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:44.052156 7239 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:44.052342 7239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:44.052531 7239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.5
I20240503 14:12:44.062309 7239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.186.5
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.5
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:44.063787 7239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:44.065732 7239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:44.079389 7244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:44.080108 7245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:44.080870 7247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:44.081772 7239 server_base.cc:1034] running on GCE node
I20240503 14:12:44.082417 7239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:44.083391 7239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:44.084617 7239 hybrid_clock.cc:648] HybridClock initialized: now 1714745564084552 us; error 58 us; skew 500 ppm
I20240503 14:12:44.084986 7239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:44.087424 7239 webserver.cc:460] Webserver started at http://127.0.186.5:45981/ using document root <none> and password file <none>
I20240503 14:12:44.088214 7239 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:44.088330 7239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:44.088658 7239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:44.091420 7239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5b7e36bf354f480c90859cb8867b59cf"
format_stamp: "Formatted at 2024-05-03 14:12:44 on dist-test-slave-mh2c"
I20240503 14:12:44.092293 7239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5b7e36bf354f480c90859cb8867b59cf"
format_stamp: "Formatted at 2024-05-03 14:12:44 on dist-test-slave-mh2c"
I20240503 14:12:44.097669 7239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:12:44.101449 7253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:44.102933 7239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:44.103107 7239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5b7e36bf354f480c90859cb8867b59cf"
format_stamp: "Formatted at 2024-05-03 14:12:44 on dist-test-slave-mh2c"
I20240503 14:12:44.103336 7239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:44.123486 7239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:44.124452 7239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:44.124737 7239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:44.125715 7239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:44.127200 7239 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:44.127300 7239 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:44.127419 7239 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:44.127492 7239 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:44.159806 7239 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.5:36609
I20240503 14:12:44.159866 7365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.5:36609 every 8 connection(s)
I20240503 14:12:44.161129 7239 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:12:44.161978 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 7239
I20240503 14:12:44.162163 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 14:12:44.165747 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.6:0
--local_ip_for_outbound_sockets=127.0.186.6
--webserver_interface=127.0.186.6
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:44.176733 7366 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:12:44.177049 7366 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:44.177896 7366 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:12:44.179483 6659 ts_manager.cc:194] Registered new tserver with Master: 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:12:44.180752 6659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.5:34969
I20240503 14:12:44.247257 7106 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
W20240503 14:12:44.512465 7369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:44.512768 7369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:44.512861 7369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:44.520694 7369 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:44.520828 7369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:44.520975 7369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.6
I20240503 14:12:44.530862 7369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.6:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.186.6
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.6
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:44.532224 7369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:44.534049 7369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:44.547376 7377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:44.548028 7374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:44.548722 7375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:44.549712 7369 server_base.cc:1034] running on GCE node
I20240503 14:12:44.550415 7369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:44.551470 7369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:44.552682 7369 hybrid_clock.cc:648] HybridClock initialized: now 1714745564552621 us; error 61 us; skew 500 ppm
I20240503 14:12:44.553033 7369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:44.555408 7369 webserver.cc:460] Webserver started at http://127.0.186.6:46767/ using document root <none> and password file <none>
I20240503 14:12:44.556286 7369 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:44.556411 7369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:44.556733 7369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:44.559633 7369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "f03dc37fe3234033bc665bed4f1b51f5"
format_stamp: "Formatted at 2024-05-03 14:12:44 on dist-test-slave-mh2c"
I20240503 14:12:44.560485 7369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "f03dc37fe3234033bc665bed4f1b51f5"
format_stamp: "Formatted at 2024-05-03 14:12:44 on dist-test-slave-mh2c"
I20240503 14:12:44.565961 7369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 14:12:44.569641 7383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:44.571084 7369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:44.571264 7369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f03dc37fe3234033bc665bed4f1b51f5"
format_stamp: "Formatted at 2024-05-03 14:12:44 on dist-test-slave-mh2c"
I20240503 14:12:44.571483 7369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:44.597030 7369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:44.597918 7369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:44.598206 7369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:44.599187 7369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:44.600803 7369 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:44.600900 7369 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:44.601068 7369 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:44.601151 7369 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:44.635735 7369 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.6:33971
I20240503 14:12:44.635788 7495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.6:33971 every 8 connection(s)
I20240503 14:12:44.637048 7369 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:12:44.638510 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 7369
I20240503 14:12:44.638664 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 14:12:44.641940 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.7:0
--local_ip_for_outbound_sockets=127.0.186.7
--webserver_interface=127.0.186.7
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:44.651172 7496 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:12:44.651504 7496 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:44.652377 7496 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:12:44.653796 6659 ts_manager.cc:194] Registered new tserver with Master: f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:12:44.654572 6659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.6:38395
I20240503 14:12:44.716207 7236 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
W20240503 14:12:44.980314 7499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:44.980654 7499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:44.980808 7499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:44.988904 7499 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:44.989069 7499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:44.989248 7499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.7
I20240503 14:12:44.998566 7499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.7:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.186.7
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.7
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:45.000061 7499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:45.001953 7499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:45.015164 7504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:45.015772 7505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:45.016387 7507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:45.017107 7499 server_base.cc:1034] running on GCE node
I20240503 14:12:45.017786 7499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:45.018824 7499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:45.020042 7499 hybrid_clock.cc:648] HybridClock initialized: now 1714745565019978 us; error 56 us; skew 500 ppm
I20240503 14:12:45.020396 7499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:45.026991 7499 webserver.cc:460] Webserver started at http://127.0.186.7:33785/ using document root <none> and password file <none>
I20240503 14:12:45.027874 7499 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:45.028002 7499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:45.028343 7499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:45.031133 7499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "3efdbd3506274dba846ae40404f0572c"
format_stamp: "Formatted at 2024-05-03 14:12:45 on dist-test-slave-mh2c"
I20240503 14:12:45.032009 7499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "3efdbd3506274dba846ae40404f0572c"
format_stamp: "Formatted at 2024-05-03 14:12:45 on dist-test-slave-mh2c"
I20240503 14:12:45.037503 7499 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:45.041217 7513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:45.042640 7499 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:45.042804 7499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "3efdbd3506274dba846ae40404f0572c"
format_stamp: "Formatted at 2024-05-03 14:12:45 on dist-test-slave-mh2c"
I20240503 14:12:45.043005 7499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:45.087055 7499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:45.087996 7499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:45.088290 7499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:45.089285 7499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:45.090806 7499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:45.090904 7499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:45.091032 7499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:45.091106 7499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:45.123179 7499 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.7:42783
I20240503 14:12:45.123261 7625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.7:42783 every 8 connection(s)
I20240503 14:12:45.124562 7499 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:12:45.134913 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 7499
I20240503 14:12:45.135107 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 14:12:45.137662 7626 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:12:45.138031 7626 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:45.138520 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.8:0
--local_ip_for_outbound_sockets=127.0.186.8
--webserver_interface=127.0.186.8
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:12:45.139256 7626 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:12:45.141026 6658 ts_manager.cc:194] Registered new tserver with Master: 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:12:45.142340 6658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.7:35631
I20240503 14:12:45.183382 7366 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
W20240503 14:12:45.490587 7629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:45.490900 7629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:12:45.491001 7629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:45.499065 7629 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:12:45.499212 7629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:12:45.499389 7629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.8
I20240503 14:12:45.509644 7629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.8:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.186.8
--webserver_port=0
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.8
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:12:45.511037 7629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:12:45.512923 7629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:12:45.526535 7637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:45.526947 7634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:12:45.527650 7635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:12:45.528401 7629 server_base.cc:1034] running on GCE node
I20240503 14:12:45.529047 7629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:12:45.530038 7629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:12:45.531260 7629 hybrid_clock.cc:648] HybridClock initialized: now 1714745565531216 us; error 36 us; skew 500 ppm
I20240503 14:12:45.531632 7629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:12:45.534036 7629 webserver.cc:460] Webserver started at http://127.0.186.8:33785/ using document root <none> and password file <none>
I20240503 14:12:45.534790 7629 fs_manager.cc:359] Metadata directory not provided
I20240503 14:12:45.534890 7629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:12:45.535212 7629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 14:12:45.537981 7629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "ea801601713a4d55aee0de9e3473e0bc"
format_stamp: "Formatted at 2024-05-03 14:12:45 on dist-test-slave-mh2c"
I20240503 14:12:45.538851 7629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "ea801601713a4d55aee0de9e3473e0bc"
format_stamp: "Formatted at 2024-05-03 14:12:45 on dist-test-slave-mh2c"
I20240503 14:12:45.544765 7629 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240503 14:12:45.548753 7643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:45.550400 7629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:12:45.550587 7629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ea801601713a4d55aee0de9e3473e0bc"
format_stamp: "Formatted at 2024-05-03 14:12:45 on dist-test-slave-mh2c"
I20240503 14:12:45.550805 7629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:12:45.591285 7629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:12:45.592229 7629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:12:45.592499 7629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:12:45.593502 7629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:12:45.595024 7629 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:12:45.595122 7629 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:45.595235 7629 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:12:45.595314 7629 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:12:45.627550 7629 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.8:36653
I20240503 14:12:45.627645 7755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.8:36653 every 8 connection(s)
I20240503 14:12:45.628875 7629 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:12:45.631472 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 7629
I20240503 14:12:45.631695 744 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 14:12:45.642997 7756 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:12:45.643312 7756 heartbeater.cc:464] Registering TS with master...
I20240503 14:12:45.644207 7756 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:12:45.645665 6658 ts_manager.cc:194] Registered new tserver with Master: ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:12:45.646476 6658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.8:48819
I20240503 14:12:45.648403 744 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:12:45.656994 7496 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
I20240503 14:12:45.665604 744 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 14:12:45.665760 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 7239
I20240503 14:12:45.674289 7361 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:45.761988 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 7239
I20240503 14:12:45.778622 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 7369
I20240503 14:12:45.786861 7491 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:45.869724 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 7369
I20240503 14:12:45.886045 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 7499
I20240503 14:12:45.894240 7621 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:45.975375 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 7499
I20240503 14:12:45.991539 744 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu pid 7629
I20240503 14:12:46.000387 7751 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 14:12:46.082752 744 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu with pid 7629
I20240503 14:12:49.851341 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:49.893611 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 176e442aa7f245bc882c15765160ad54 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.895293 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176e442aa7f245bc882c15765160ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.895385 7041 tablet_service.cc:1467] Processing CreateTablet for tablet bef030b74c05488a9d8f3ec0df23054a (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.895932 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef030b74c05488a9d8f3ec0df23054a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.896896 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 7ece38bbe1e5401ba4b10607ffaf9901 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.896896 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 7ece38bbe1e5401ba4b10607ffaf9901 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.897943 6910 tablet_service.cc:1467] Processing CreateTablet for tablet bef030b74c05488a9d8f3ec0df23054a (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.898341 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ece38bbe1e5401ba4b10607ffaf9901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.898387 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ece38bbe1e5401ba4b10607ffaf9901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.898890 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef030b74c05488a9d8f3ec0df23054a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.898918 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 176e442aa7f245bc882c15765160ad54 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.899804 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176e442aa7f245bc882c15765160ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.907089 6779 tablet_service.cc:1467] Processing CreateTablet for tablet bef030b74c05488a9d8f3ec0df23054a (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.908124 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 176e442aa7f245bc882c15765160ad54 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.909718 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 176e442aa7f245bc882c15765160ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.908627 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 7ece38bbe1e5401ba4b10607ffaf9901 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:49.914036 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ece38bbe1e5401ba4b10607ffaf9901. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.915398 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bef030b74c05488a9d8f3ec0df23054a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:49.922894 7787 tablet_bootstrap.cc:492] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:49.922888 7786 tablet_bootstrap.cc:492] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:49.926728 7786 tablet_bootstrap.cc:654] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.922147 7785 tablet_bootstrap.cc:492] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:49.928352 7787 tablet_bootstrap.cc:654] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.929826 7787 log.cc:824] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:49.932408 7785 tablet_bootstrap.cc:654] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.933866 7786 log.cc:824] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:49.935389 7785 log.cc:824] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:49.939836 7789 tablet_bootstrap.cc:492] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:49.940850 7786 tablet_bootstrap.cc:492] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:49.940850 7787 tablet_bootstrap.cc:492] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:49.941169 7786 ts_tablet_manager.cc:1367] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.001s
I20240503 14:12:49.941169 7787 ts_tablet_manager.cc:1367] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.006s
I20240503 14:12:49.941440 7785 tablet_bootstrap.cc:492] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:49.941723 7785 ts_tablet_manager.cc:1367] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.002s
I20240503 14:12:49.944470 7789 tablet_bootstrap.cc:654] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.945962 7789 log.cc:824] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: Log is configured to *not* fsync() on all Append() calls
I20240503 14:12:49.950852 7786 raft_consensus.cc:348] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.951388 7786 raft_consensus.cc:374] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.951504 7786 raft_consensus.cc:729] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.952044 7787 raft_consensus.cc:348] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.952375 7786 consensus_queue.cc:260] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.952567 7787 raft_consensus.cc:374] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.952675 7787 raft_consensus.cc:729] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.953414 7787 consensus_queue.cc:260] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.953398 7785 raft_consensus.cc:348] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.953840 7785 raft_consensus.cc:374] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.953950 7785 raft_consensus.cc:729] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.954588 7786 ts_tablet_manager.cc:1398] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.013s user 0.007s sys 0.006s
I20240503 14:12:49.954546 7785 consensus_queue.cc:260] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.955019 7786 tablet_bootstrap.cc:492] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:49.957381 7786 tablet_bootstrap.cc:654] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.962407 7786 tablet_bootstrap.cc:492] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:49.962635 7786 ts_tablet_manager.cc:1367] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240503 14:12:49.964040 7786 raft_consensus.cc:348] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.964362 7786 raft_consensus.cc:374] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.964483 7786 raft_consensus.cc:729] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.964924 7786 consensus_queue.cc:260] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.965590 7786 ts_tablet_manager.cc:1398] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:12:49.967608 7785 ts_tablet_manager.cc:1398] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.026s user 0.018s sys 0.006s
I20240503 14:12:49.967716 7789 tablet_bootstrap.cc:492] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:49.968012 7789 ts_tablet_manager.cc:1367] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.003s
I20240503 14:12:49.968116 7785 tablet_bootstrap.cc:492] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:49.970113 7785 tablet_bootstrap.cc:654] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.971429 7787 ts_tablet_manager.cc:1398] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.030s user 0.012s sys 0.010s
I20240503 14:12:49.974032 7787 tablet_bootstrap.cc:492] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:49.975811 7787 tablet_bootstrap.cc:654] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.979357 7787 tablet_bootstrap.cc:492] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:49.979357 7785 tablet_bootstrap.cc:492] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:49.979542 7785 ts_tablet_manager.cc:1367] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20240503 14:12:49.979281 7789 raft_consensus.cc:348] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.979934 7787 ts_tablet_manager.cc:1367] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:12:49.980222 7789 raft_consensus.cc:374] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.980180 7794 raft_consensus.cc:482] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:49.980621 7789 raft_consensus.cc:729] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.980770 7794 raft_consensus.cc:504] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.981878 7785 raft_consensus.cc:348] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.981931 7787 raft_consensus.cc:348] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.982143 7785 raft_consensus.cc:374] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.982143 7787 raft_consensus.cc:374] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.982251 7787 raft_consensus.cc:729] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.982261 7785 raft_consensus.cc:729] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.981986 7789 consensus_queue.cc:260] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.982527 7787 consensus_queue.cc:260] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.982614 7785 consensus_queue.cc:260] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.983186 7787 ts_tablet_manager.cc:1398] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240503 14:12:49.983382 7107 tablet.cc:2369] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:49.986999 7794 leader_election.cc:288] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:12:49.988899 7785 ts_tablet_manager.cc:1398] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:12:49.989955 7789 ts_tablet_manager.cc:1398] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.022s user 0.014s sys 0.001s
I20240503 14:12:49.990437 7789 tablet_bootstrap.cc:492] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:49.992363 7789 tablet_bootstrap.cc:654] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:49.994712 7789 tablet_bootstrap.cc:492] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:49.994886 7789 ts_tablet_manager.cc:1367] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:49.996206 7789 raft_consensus.cc:348] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.996480 7789 raft_consensus.cc:374] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:49.996603 7789 raft_consensus.cc:729] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:49.996884 7789 consensus_queue.cc:260] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:49.997534 7789 ts_tablet_manager.cc:1398] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:49.997869 7789 tablet_bootstrap.cc:492] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:49.999367 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef030b74c05488a9d8f3ec0df23054a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:50.000006 7789 tablet_bootstrap.cc:654] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:50.000036 7061 raft_consensus.cc:2454] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
W20240503 14:12:50.000873 6977 tablet.cc:2369] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:50.002013 6867 leader_election.cc:302] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:12:50.002599 7794 raft_consensus.cc:2781] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:50.002787 7794 raft_consensus.cc:482] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:50.002919 7794 raft_consensus.cc:3037] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.003535 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef030b74c05488a9d8f3ec0df23054a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:50.004199 6800 raft_consensus.cc:2454] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:12:50.005550 7789 tablet_bootstrap.cc:492] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:50.005743 7789 ts_tablet_manager.cc:1367] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 14:12:50.006470 7794 raft_consensus.cc:504] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.006605 7789 raft_consensus.cc:348] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.006870 7789 raft_consensus.cc:374] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:50.006960 7789 raft_consensus.cc:729] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:50.007356 7794 leader_election.cc:288] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:12:50.007252 7789 consensus_queue.cc:260] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.007886 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef030b74c05488a9d8f3ec0df23054a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:50.008004 7789 ts_tablet_manager.cc:1398] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:50.007954 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bef030b74c05488a9d8f3ec0df23054a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:50.008072 6800 raft_consensus.cc:3037] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.008184 7061 raft_consensus.cc:3037] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.010965 7061 raft_consensus.cc:2454] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:12:50.011029 6800 raft_consensus.cc:2454] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:12:50.011507 6865 leader_election.cc:302] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:50.011946 7794 raft_consensus.cc:2781] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:50.012166 7794 raft_consensus.cc:686] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:12:50.012661 7794 consensus_queue.cc:237] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.017848 6658 catalog_manager.cc:5484] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
W20240503 14:12:50.028561 6846 tablet.cc:2369] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:50.048664 7799 raft_consensus.cc:482] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:50.048877 7799 raft_consensus.cc:504] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.050586 7799 leader_election.cc:288] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:50.057704 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176e442aa7f245bc882c15765160ad54" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:50.057976 7061 raft_consensus.cc:2454] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:50.057885 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176e442aa7f245bc882c15765160ad54" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:50.058408 7191 raft_consensus.cc:2454] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:50.058487 6736 leader_election.cc:302] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:50.058910 7799 raft_consensus.cc:2781] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:50.059037 7799 raft_consensus.cc:482] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:50.059144 7799 raft_consensus.cc:3037] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.061214 7799 raft_consensus.cc:504] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.061856 7799 leader_election.cc:288] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:50.062353 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176e442aa7f245bc882c15765160ad54" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:50.062486 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "176e442aa7f245bc882c15765160ad54" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:50.062616 7061 raft_consensus.cc:3037] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.062685 7191 raft_consensus.cc:3037] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.065629 7061 raft_consensus.cc:2454] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:50.065704 7191 raft_consensus.cc:2454] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:50.066102 6735 leader_election.cc:302] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:50.066366 7799 raft_consensus.cc:2781] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:50.066756 7799 raft_consensus.cc:686] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:12:50.067351 7799 consensus_queue.cc:237] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.072037 6658 catalog_manager.cc:5484] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:50.125178 7799 raft_consensus.cc:482] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:50.125424 7799 raft_consensus.cc:504] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.126200 7799 leader_election.cc:288] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:50.126745 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ece38bbe1e5401ba4b10607ffaf9901" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:50.126986 7191 raft_consensus.cc:2454] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:50.127395 6735 leader_election.cc:302] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:50.127701 7799 raft_consensus.cc:2781] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:50.127786 7799 raft_consensus.cc:482] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:50.127892 7799 raft_consensus.cc:3037] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.129794 7799 raft_consensus.cc:504] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.130455 7799 leader_election.cc:288] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:50.130837 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ece38bbe1e5401ba4b10607ffaf9901" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:50.131090 7191 raft_consensus.cc:3037] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.132738 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ece38bbe1e5401ba4b10607ffaf9901" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:50.132704 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ece38bbe1e5401ba4b10607ffaf9901" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:50.133230 7191 raft_consensus.cc:2454] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:50.133226 6931 raft_consensus.cc:2454] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:50.133656 6735 leader_election.cc:302] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:50.134162 7799 raft_consensus.cc:2781] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:50.134408 7799 raft_consensus.cc:686] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:12:50.134711 7799 consensus_queue.cc:237] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:50.137599 6658 catalog_manager.cc:5484] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
W20240503 14:12:50.200974 7237 tablet.cc:2369] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:12:50.402868 7794 consensus_queue.cc:1035] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:50.411055 7794 consensus_queue.cc:1035] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:50.458551 7799 consensus_queue.cc:1035] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:50.467651 7799 consensus_queue.cc:1035] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:50.550505 7811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:50.550892 7811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:50.559012 7811 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:50.576588 7808 consensus_queue.cc:1035] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:50.585038 6930 raft_consensus.cc:3037] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:50.589489 7799 consensus_queue.cc:1035] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 235.168 ms
time per row: 0.117584 ms
-----------------------------------------------------
Suppressions used:
count bytes template
16 4220 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:51.896837 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:51.906914 7040 tablet_service.cc:1467] Processing CreateTablet for tablet f56e8ea68eb94475838da245b5f060cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.907217 6910 tablet_service.cc:1467] Processing CreateTablet for tablet f56e8ea68eb94475838da245b5f060cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.907459 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56e8ea68eb94475838da245b5f060cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.907732 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56e8ea68eb94475838da245b5f060cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.908411 7170 tablet_service.cc:1467] Processing CreateTablet for tablet f56e8ea68eb94475838da245b5f060cc (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.908731 7041 tablet_service.cc:1467] Processing CreateTablet for tablet a44ed92d65114bfda2e0f38054b17ada (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.908828 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56e8ea68eb94475838da245b5f060cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.909112 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44ed92d65114bfda2e0f38054b17ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.914137 7872 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:51.914265 7039 tablet_service.cc:1467] Processing CreateTablet for tablet dcb23bc5b0a44da7989aa828826ece07 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.914426 6779 tablet_service.cc:1467] Processing CreateTablet for tablet dcb23bc5b0a44da7989aa828826ece07 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.914772 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb23bc5b0a44da7989aa828826ece07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.915030 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb23bc5b0a44da7989aa828826ece07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.916600 6911 tablet_service.cc:1467] Processing CreateTablet for tablet a44ed92d65114bfda2e0f38054b17ada (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.917022 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44ed92d65114bfda2e0f38054b17ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.919930 7171 tablet_service.cc:1467] Processing CreateTablet for tablet a44ed92d65114bfda2e0f38054b17ada (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.920446 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44ed92d65114bfda2e0f38054b17ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.921481 6909 tablet_service.cc:1467] Processing CreateTablet for tablet dcb23bc5b0a44da7989aa828826ece07 (DEFAULT_TABLE table=rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:51.921928 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb23bc5b0a44da7989aa828826ece07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:51.932070 7872 tablet_bootstrap.cc:654] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.935896 7875 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:51.937278 7874 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:51.938872 7875 tablet_bootstrap.cc:654] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.940090 7874 tablet_bootstrap.cc:654] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.941498 7873 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:51.943254 7873 tablet_bootstrap.cc:654] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.943977 7875 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:51.944149 7875 ts_tablet_manager.cc:1367] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:12:51.945169 7875 raft_consensus.cc:348] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:51.945473 7875 raft_consensus.cc:374] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.945585 7875 raft_consensus.cc:729] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.945881 7875 consensus_queue.cc:260] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:51.946502 7875 ts_tablet_manager.cc:1398] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:51.947858 7872 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:51.947911 7874 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:51.948056 7874 ts_tablet_manager.cc:1367] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 14:12:51.948099 7872 ts_tablet_manager.cc:1367] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.005s
I20240503 14:12:51.948846 7874 raft_consensus.cc:348] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.949080 7874 raft_consensus.cc:374] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.949175 7874 raft_consensus.cc:729] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.948906 7872 raft_consensus.cc:348] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.949335 7872 raft_consensus.cc:374] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.949470 7872 raft_consensus.cc:729] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.949465 7874 consensus_queue.cc:260] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.949858 7872 consensus_queue.cc:260] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.952944 7873 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:51.953128 7873 ts_tablet_manager.cc:1367] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240503 14:12:51.952688 7874 ts_tablet_manager.cc:1398] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240503 14:12:51.953455 7872 ts_tablet_manager.cc:1398] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240503 14:12:51.953532 7874 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:51.953812 7872 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:51.953933 7873 raft_consensus.cc:348] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.954172 7873 raft_consensus.cc:374] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.954265 7873 raft_consensus.cc:729] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.954531 7873 consensus_queue.cc:260] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.955096 7873 ts_tablet_manager.cc:1398] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:51.955303 7874 tablet_bootstrap.cc:654] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.955379 7873 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:51.957211 7873 tablet_bootstrap.cc:654] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.958103 7872 tablet_bootstrap.cc:654] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.964907 7874 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:51.965129 7874 ts_tablet_manager.cc:1367] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20240503 14:12:51.966244 7874 raft_consensus.cc:348] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.966535 7874 raft_consensus.cc:374] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.966639 7874 raft_consensus.cc:729] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.966825 7872 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:51.967006 7872 ts_tablet_manager.cc:1367] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240503 14:12:51.967006 7873 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:51.966902 7874 consensus_queue.cc:260] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.967167 7873 ts_tablet_manager.cc:1367] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 14:12:51.967448 7874 ts_tablet_manager.cc:1398] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20240503 14:12:51.967790 7874 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:51.967788 7872 raft_consensus.cc:348] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.967999 7872 raft_consensus.cc:374] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.968070 7872 raft_consensus.cc:729] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.968016 7873 raft_consensus.cc:348] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.968227 7873 raft_consensus.cc:374] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.968294 7873 raft_consensus.cc:729] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.968351 7872 consensus_queue.cc:260] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.968583 7873 consensus_queue.cc:260] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.970125 7873 ts_tablet_manager.cc:1398] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:51.970425 7873 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:51.969409 7874 tablet_bootstrap.cc:654] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.971228 7872 ts_tablet_manager.cc:1398] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:51.974370 7873 tablet_bootstrap.cc:654] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:51.976212 7873 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:51.976263 7874 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:51.976369 7873 ts_tablet_manager.cc:1367] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:12:51.976394 7874 ts_tablet_manager.cc:1367] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 14:12:51.976948 7880 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:51.977066 7873 raft_consensus.cc:348] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:51.977080 7874 raft_consensus.cc:348] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:51.977278 7873 raft_consensus.cc:374] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.977137 7880 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:51.977310 7874 raft_consensus.cc:374] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:51.977365 7873 raft_consensus.cc:729] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.977401 7874 raft_consensus.cc:729] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:51.977622 7873 consensus_queue.cc:260] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:51.977653 7874 consensus_queue.cc:260] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:51.978124 7873 ts_tablet_manager.cc:1398] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:51.979631 7874 ts_tablet_manager.cc:1398] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:51.983875 7880 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:51.995819 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44ed92d65114bfda2e0f38054b17ada" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:51.995819 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44ed92d65114bfda2e0f38054b17ada" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:51.996098 7191 raft_consensus.cc:2454] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:51.996098 6930 raft_consensus.cc:2454] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:51.996680 6995 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:12:51.997076 7880 raft_consensus.cc:2781] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:51.997212 7880 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:51.997305 7880 raft_consensus.cc:3037] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:51.999397 7880 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:52.000087 7880 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:52.000620 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44ed92d65114bfda2e0f38054b17ada" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:52.000699 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a44ed92d65114bfda2e0f38054b17ada" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:52.000845 6930 raft_consensus.cc:3037] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:52.000891 7191 raft_consensus.cc:3037] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:52.003808 6930 raft_consensus.cc:2454] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:52.003808 7191 raft_consensus.cc:2454] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:52.004272 6996 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:52.004591 7880 raft_consensus.cc:2781] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:52.004758 7880 raft_consensus.cc:686] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:12:52.005180 7880 consensus_queue.cc:237] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:52.008808 6659 catalog_manager.cc:5484] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:52.050541 7867 raft_consensus.cc:482] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:52.050729 7867 raft_consensus.cc:504] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:52.051525 7867 leader_election.cc:288] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:52.052151 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56e8ea68eb94475838da245b5f060cc" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:52.052435 7060 raft_consensus.cc:2454] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:12:52.052887 6867 leader_election.cc:302] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:12:52.053189 7867 raft_consensus.cc:2781] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:52.053318 7867 raft_consensus.cc:482] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:52.053450 7867 raft_consensus.cc:3037] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:52.055488 7867 raft_consensus.cc:504] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:52.056161 7867 leader_election.cc:288] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:52.056680 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56e8ea68eb94475838da245b5f060cc" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:52.056900 7060 raft_consensus.cc:3037] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:52.058085 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56e8ea68eb94475838da245b5f060cc" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:52.058328 7191 raft_consensus.cc:2454] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:12:52.058245 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f56e8ea68eb94475838da245b5f060cc" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:52.058542 7190 raft_consensus.cc:3037] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:52.059535 7060 raft_consensus.cc:2454] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:12:52.059969 6867 leader_election.cc:302] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:12:52.060292 7867 raft_consensus.cc:2781] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:52.060691 7867 raft_consensus.cc:686] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:12:52.060972 7867 consensus_queue.cc:237] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:52.061115 7190 raft_consensus.cc:2454] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:12:52.063803 6659 catalog_manager.cc:5484] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:52.081348 7818 raft_consensus.cc:482] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:52.081538 7818 raft_consensus.cc:504] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:52.082144 7818 leader_election.cc:288] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:52.082710 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb23bc5b0a44da7989aa828826ece07" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:52.082710 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb23bc5b0a44da7989aa828826ece07" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:52.082929 6930 raft_consensus.cc:2454] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:52.082957 7060 raft_consensus.cc:2454] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:52.083402 6734 leader_election.cc:302] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:52.083710 7818 raft_consensus.cc:2781] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:52.083807 7818 raft_consensus.cc:482] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:52.083871 7818 raft_consensus.cc:3037] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:52.085884 7818 raft_consensus.cc:504] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:52.086521 7818 leader_election.cc:288] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:52.086980 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb23bc5b0a44da7989aa828826ece07" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:52.086992 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcb23bc5b0a44da7989aa828826ece07" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:52.087178 7060 raft_consensus.cc:3037] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:52.087191 6930 raft_consensus.cc:3037] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:52.089074 7060 raft_consensus.cc:2454] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:52.089074 6930 raft_consensus.cc:2454] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:52.089485 6734 leader_election.cc:302] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:52.089818 7818 raft_consensus.cc:2781] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:52.090148 7818 raft_consensus.cc:686] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:12:52.090426 7818 consensus_queue.cc:237] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:52.092964 6659 catalog_manager.cc:5484] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:52.477700 7887 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:52.478016 7898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:52.478479 7898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:12:52.479627 7887 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:52.495287 7898 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:52.496768 7867 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:52.502997 7867 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 14:12:52.554672 7818 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:52.560539 7818 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 201.213 ms
time per row: 0.100607 ms
I20240503 14:12:52.866642 6656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:52.876135 7170 tablet_service.cc:1467] Processing CreateTablet for tablet de33325421fb42f982d44ca21e00f766 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.876683 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de33325421fb42f982d44ca21e00f766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.876771 6779 tablet_service.cc:1467] Processing CreateTablet for tablet de33325421fb42f982d44ca21e00f766 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.877090 6778 tablet_service.cc:1467] Processing CreateTablet for tablet d6d31c9abf474fff9a7efd9a96679ad4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.877791 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 58003e797fac46ef8b9689728eebbd43 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.877833 7171 tablet_service.cc:1467] Processing CreateTablet for tablet d6d31c9abf474fff9a7efd9a96679ad4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.878192 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d31c9abf474fff9a7efd9a96679ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.878240 7041 tablet_service.cc:1467] Processing CreateTablet for tablet d6d31c9abf474fff9a7efd9a96679ad4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.878631 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d31c9abf474fff9a7efd9a96679ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.878541 6910 tablet_service.cc:1467] Processing CreateTablet for tablet de33325421fb42f982d44ca21e00f766 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.878952 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de33325421fb42f982d44ca21e00f766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.877203 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de33325421fb42f982d44ca21e00f766. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.884465 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d31c9abf474fff9a7efd9a96679ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.884728 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58003e797fac46ef8b9689728eebbd43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.886500 7963 tablet_bootstrap.cc:492] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:52.889009 7963 tablet_bootstrap.cc:654] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.896883 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 58003e797fac46ef8b9689728eebbd43 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.897406 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58003e797fac46ef8b9689728eebbd43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.898298 7963 tablet_bootstrap.cc:492] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:52.898452 7963 ts_tablet_manager.cc:1367] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 14:12:52.899320 7963 raft_consensus.cc:348] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.899535 7963 raft_consensus.cc:374] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.899648 7963 raft_consensus.cc:729] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.899982 7963 consensus_queue.cc:260] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.900259 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 58003e797fac46ef8b9689728eebbd43 (DEFAULT_TABLE table=rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:52.900719 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58003e797fac46ef8b9689728eebbd43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:52.900835 7963 ts_tablet_manager.cc:1398] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:52.901667 7963 tablet_bootstrap.cc:492] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:52.903170 7903 raft_consensus.cc:482] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:52.903322 7903 raft_consensus.cc:504] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.905305 7903 leader_election.cc:288] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:52.906306 7967 tablet_bootstrap.cc:492] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:52.909298 7966 tablet_bootstrap.cc:492] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:52.909687 7964 tablet_bootstrap.cc:492] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:52.911170 7966 tablet_bootstrap.cc:654] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.912079 7967 tablet_bootstrap.cc:654] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.912354 7964 tablet_bootstrap.cc:654] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.916949 7963 tablet_bootstrap.cc:654] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.922863 7963 tablet_bootstrap.cc:492] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:52.923048 7963 ts_tablet_manager.cc:1367] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20240503 14:12:52.923141 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33325421fb42f982d44ca21e00f766" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:52.923959 7963 raft_consensus.cc:348] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:52.924202 7963 raft_consensus.cc:374] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.924278 7963 raft_consensus.cc:729] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.924662 7963 consensus_queue.cc:260] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
W20240503 14:12:52.925139 7125 leader_election.cc:341] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:52.926388 7963 ts_tablet_manager.cc:1398] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:52.926700 7963 tablet_bootstrap.cc:492] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:52.926827 7964 tablet_bootstrap.cc:492] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:52.926980 7964 ts_tablet_manager.cc:1367] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20240503 14:12:52.927876 7964 raft_consensus.cc:348] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:52.928085 7964 raft_consensus.cc:374] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.928177 7964 raft_consensus.cc:729] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.928439 7963 tablet_bootstrap.cc:654] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.928488 7964 consensus_queue.cc:260] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:52.929024 7964 ts_tablet_manager.cc:1398] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:52.929047 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33325421fb42f982d44ca21e00f766" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:52.930199 7966 tablet_bootstrap.cc:492] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:52.930369 7966 ts_tablet_manager.cc:1367] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.008s
W20240503 14:12:52.931071 7125 leader_election.cc:341] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:52.931401 7125 leader_election.cc:302] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13
I20240503 14:12:52.932279 7966 raft_consensus.cc:348] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:52.932502 7966 raft_consensus.cc:374] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.932590 7966 raft_consensus.cc:729] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.932565 7903 raft_consensus.cc:2726] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:52.932830 7966 consensus_queue.cc:260] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:52.933405 7966 ts_tablet_manager.cc:1398] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 14:12:52.933743 7966 tablet_bootstrap.cc:492] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:52.934276 7967 tablet_bootstrap.cc:492] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:52.934430 7967 ts_tablet_manager.cc:1367] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20240503 14:12:52.935420 7966 tablet_bootstrap.cc:654] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.935413 7967 raft_consensus.cc:348] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.935751 7967 raft_consensus.cc:374] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.935859 7967 raft_consensus.cc:729] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.936123 7967 consensus_queue.cc:260] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.936565 7963 tablet_bootstrap.cc:492] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:52.936726 7963 ts_tablet_manager.cc:1367] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 14:12:52.937515 7963 raft_consensus.cc:348] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.937721 7963 raft_consensus.cc:374] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.937806 7963 raft_consensus.cc:729] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.937891 7967 ts_tablet_manager.cc:1398] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:52.938035 7963 consensus_queue.cc:260] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.938161 7967 tablet_bootstrap.cc:492] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:52.938572 7963 ts_tablet_manager.cc:1398] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:52.940184 7967 tablet_bootstrap.cc:654] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.940570 7966 tablet_bootstrap.cc:492] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:52.940759 7966 ts_tablet_manager.cc:1367] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.001s
I20240503 14:12:52.941864 7966 raft_consensus.cc:348] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.942116 7966 raft_consensus.cc:374] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.942234 7966 raft_consensus.cc:729] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.942531 7967 tablet_bootstrap.cc:492] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:52.942533 7966 consensus_queue.cc:260] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.942725 7967 ts_tablet_manager.cc:1367] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 14:12:52.943432 7966 ts_tablet_manager.cc:1398] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:12:52.943764 7966 tablet_bootstrap.cc:492] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:52.943742 7967 raft_consensus.cc:348] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.943958 7967 raft_consensus.cc:374] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.944036 7967 raft_consensus.cc:729] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.944319 7967 consensus_queue.cc:260] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.944885 7967 ts_tablet_manager.cc:1398] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:52.945423 7966 tablet_bootstrap.cc:654] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:52.947460 7966 tablet_bootstrap.cc:492] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:52.947633 7966 ts_tablet_manager.cc:1367] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:52.948271 7966 raft_consensus.cc:348] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.948477 7966 raft_consensus.cc:374] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:52.948556 7966 raft_consensus.cc:729] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:52.948717 7966 consensus_queue.cc:260] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:52.949144 7966 ts_tablet_manager.cc:1398] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:12:53.071362 7896 raft_consensus.cc:482] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:53.071617 7896 raft_consensus.cc:504] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:53.072415 7896 leader_election.cc:288] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:53.072993 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d31c9abf474fff9a7efd9a96679ad4" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:53.073004 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d31c9abf474fff9a7efd9a96679ad4" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:53.073213 7060 raft_consensus.cc:2454] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:53.073269 7190 raft_consensus.cc:2454] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:53.073730 6736 leader_election.cc:302] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:53.073999 7896 raft_consensus.cc:2781] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:53.074119 7896 raft_consensus.cc:482] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:53.074189 7896 raft_consensus.cc:3037] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.076105 7896 raft_consensus.cc:504] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:53.076673 7896 leader_election.cc:288] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:53.077221 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d31c9abf474fff9a7efd9a96679ad4" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:53.077219 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6d31c9abf474fff9a7efd9a96679ad4" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:53.077418 7060 raft_consensus.cc:3037] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.077435 7190 raft_consensus.cc:3037] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.079458 7060 raft_consensus.cc:2454] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:53.079574 7190 raft_consensus.cc:2454] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:53.079998 6736 leader_election.cc:302] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:53.080354 7896 raft_consensus.cc:2781] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:53.080471 7896 raft_consensus.cc:686] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:12:53.080735 7896 consensus_queue.cc:237] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:53.083806 6658 catalog_manager.cc:5484] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:53.157590 7903 raft_consensus.cc:482] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:53.157788 7903 raft_consensus.cc:504] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:53.158483 7903 leader_election.cc:288] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:53.159154 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58003e797fac46ef8b9689728eebbd43" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:53.159189 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58003e797fac46ef8b9689728eebbd43" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:53.159415 6800 raft_consensus.cc:2454] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:53.159448 6930 raft_consensus.cc:2454] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:53.159971 7125 leader_election.cc:302] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:53.160316 7903 raft_consensus.cc:2781] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:53.160480 7903 raft_consensus.cc:482] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:53.160585 7903 raft_consensus.cc:3037] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.162709 7903 raft_consensus.cc:504] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:53.163451 7903 leader_election.cc:288] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:53.164018 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58003e797fac46ef8b9689728eebbd43" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:53.164002 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58003e797fac46ef8b9689728eebbd43" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:53.164307 6800 raft_consensus.cc:3037] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.164324 6930 raft_consensus.cc:3037] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.167330 6800 raft_consensus.cc:2454] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:53.167308 6930 raft_consensus.cc:2454] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:53.167904 7125 leader_election.cc:302] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:53.168232 7903 raft_consensus.cc:2781] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:53.168690 7903 raft_consensus.cc:686] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:12:53.169153 7903 consensus_queue.cc:237] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:53.172724 6658 catalog_manager.cc:5484] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:53.350419 7894 raft_consensus.cc:482] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:53.350627 7894 raft_consensus.cc:504] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:53.351235 7894 leader_election.cc:288] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:12:53.351846 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33325421fb42f982d44ca21e00f766" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:53.351840 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33325421fb42f982d44ca21e00f766" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:53.352062 7190 raft_consensus.cc:2454] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:12:53.352087 6800 raft_consensus.cc:2454] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:12:53.352540 6866 leader_election.cc:302] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:53.352804 7894 raft_consensus.cc:2781] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:53.352918 7894 raft_consensus.cc:482] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:53.353009 7894 raft_consensus.cc:3037] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.354888 7894 raft_consensus.cc:504] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:53.355489 7894 leader_election.cc:288] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:12:53.356017 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33325421fb42f982d44ca21e00f766" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:53.356050 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de33325421fb42f982d44ca21e00f766" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:53.356221 7190 raft_consensus.cc:3037] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.356245 6800 raft_consensus.cc:3037] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:53.358233 6800 raft_consensus.cc:2454] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:12:53.358330 7190 raft_consensus.cc:2454] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:12:53.358690 6865 leader_election.cc:302] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:53.359031 7894 raft_consensus.cc:2781] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:53.359424 7894 raft_consensus.cc:686] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:12:53.359714 7894 consensus_queue.cc:237] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:53.362390 6658 catalog_manager.cc:5484] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:53.490528 7896 consensus_queue.cc:1035] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:53.497959 7896 consensus_queue.cc:1035] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:53.662912 7903 consensus_queue.cc:1035] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:53.668133 7979 consensus_queue.cc:1035] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:53.827576 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:53.827967 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:53.835815 7983 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 14:12:53.907172 7894 consensus_queue.cc:1035] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:53.922825 7894 consensus_queue.cc:1035] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 239.38 ms
time per row: 0.11969 ms
I20240503 14:12:54.231149 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:54.241070 6778 tablet_service.cc:1467] Processing CreateTablet for tablet ca522e5b44fd44289b4b161f27f71d20 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.241178 6911 tablet_service.cc:1467] Processing CreateTablet for tablet ca522e5b44fd44289b4b161f27f71d20 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.241581 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca522e5b44fd44289b4b161f27f71d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.241639 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca522e5b44fd44289b4b161f27f71d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.241693 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0bb82edac2464b9d618c1c80807fb2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.242069 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0bb82edac2464b9d618c1c80807fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.247045 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0bb82edac2464b9d618c1c80807fb2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.247133 7041 tablet_service.cc:1467] Processing CreateTablet for tablet ca522e5b44fd44289b4b161f27f71d20 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.247499 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0bb82edac2464b9d618c1c80807fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.247797 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 24090e6169dd44f8b47c5cd772934710 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.248291 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24090e6169dd44f8b47c5cd772934710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.252430 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca522e5b44fd44289b4b161f27f71d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.252647 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 6e0bb82edac2464b9d618c1c80807fb2 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.253046 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0bb82edac2464b9d618c1c80807fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.256776 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 24090e6169dd44f8b47c5cd772934710 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.257191 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24090e6169dd44f8b47c5cd772934710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.257627 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 24090e6169dd44f8b47c5cd772934710 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:54.258056 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24090e6169dd44f8b47c5cd772934710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:54.263830 8054 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:54.266502 8054 tablet_bootstrap.cc:654] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.270691 8055 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:54.272601 8055 tablet_bootstrap.cc:654] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.278244 8055 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:54.278414 8055 ts_tablet_manager.cc:1367] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 14:12:54.279208 8055 raft_consensus.cc:348] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.279433 8055 raft_consensus.cc:374] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.279548 8055 raft_consensus.cc:729] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.281765 8055 consensus_queue.cc:260] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.282985 8055 ts_tablet_manager.cc:1398] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:54.290839 8058 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:54.293507 8058 tablet_bootstrap.cc:654] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.302381 8055 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:54.306653 8055 tablet_bootstrap.cc:654] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.309576 8053 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:54.309885 8055 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:54.310062 8055 ts_tablet_manager.cc:1367] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:12:54.310825 8055 raft_consensus.cc:348] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.311103 8055 raft_consensus.cc:374] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.311164 8053 tablet_bootstrap.cc:654] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.311180 8055 raft_consensus.cc:729] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.311455 8055 consensus_queue.cc:260] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.312063 8055 ts_tablet_manager.cc:1398] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:54.312367 8055 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:54.313876 8055 tablet_bootstrap.cc:654] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.316308 8054 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:54.316428 8058 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:54.316460 8054 ts_tablet_manager.cc:1367] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.053s user 0.013s sys 0.031s
I20240503 14:12:54.316603 8058 ts_tablet_manager.cc:1367] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20240503 14:12:54.317274 8054 raft_consensus.cc:348] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.317495 8054 raft_consensus.cc:374] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.317588 8054 raft_consensus.cc:729] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.317548 8058 raft_consensus.cc:348] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.317750 8058 raft_consensus.cc:374] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.317839 8058 raft_consensus.cc:729] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.317850 8054 consensus_queue.cc:260] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.318099 8058 consensus_queue.cc:260] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.318404 8054 ts_tablet_manager.cc:1398] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:12:54.318782 8054 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:54.320519 8054 tablet_bootstrap.cc:654] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.321446 8058 ts_tablet_manager.cc:1398] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:12:54.321758 8058 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:54.325057 8058 tablet_bootstrap.cc:654] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.326869 8058 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:54.327041 8058 ts_tablet_manager.cc:1367] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:54.328626 8058 raft_consensus.cc:348] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.328864 8058 raft_consensus.cc:374] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.328935 8058 raft_consensus.cc:729] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.329185 8058 consensus_queue.cc:260] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.329985 8058 ts_tablet_manager.cc:1398] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:54.330073 8053 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:54.330225 8053 ts_tablet_manager.cc:1367] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.005s
I20240503 14:12:54.330376 8055 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:54.330525 8055 ts_tablet_manager.cc:1367] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240503 14:12:54.330886 8054 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:54.331066 8054 ts_tablet_manager.cc:1367] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240503 14:12:54.331089 8053 raft_consensus.cc:348] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.331339 8053 raft_consensus.cc:374] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.331286 8055 raft_consensus.cc:348] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.331429 8053 raft_consensus.cc:729] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.331497 8055 raft_consensus.cc:374] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.331578 8055 raft_consensus.cc:729] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.331835 8055 consensus_queue.cc:260] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.331786 8053 consensus_queue.cc:260] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.331897 8054 raft_consensus.cc:348] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.332113 8054 raft_consensus.cc:374] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.332182 8054 raft_consensus.cc:729] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.332408 8055 ts_tablet_manager.cc:1398] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:54.332474 8053 ts_tablet_manager.cc:1398] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:12:54.332751 8053 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:54.332499 8054 consensus_queue.cc:260] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.333652 8054 ts_tablet_manager.cc:1398] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:54.334578 8053 tablet_bootstrap.cc:654] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:54.340201 8053 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:54.340452 8053 ts_tablet_manager.cc:1367] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240503 14:12:54.341424 8053 raft_consensus.cc:348] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.341799 8053 raft_consensus.cc:374] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:54.341878 8053 raft_consensus.cc:729] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:54.342090 8053 consensus_queue.cc:260] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.342610 8053 ts_tablet_manager.cc:1398] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:12:54.352542 7909 raft_consensus.cc:482] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:54.352808 7909 raft_consensus.cc:504] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.353731 7909 leader_election.cc:288] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:12:54.354043 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:54.354317 7191 raft_consensus.cc:2454] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:54.354846 6996 leader_election.cc:302] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:54.355158 7909 raft_consensus.cc:2781] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:54.355259 7909 raft_consensus.cc:482] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:54.355360 7909 raft_consensus.cc:3037] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.357430 7909 raft_consensus.cc:504] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.358090 7909 leader_election.cc:288] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:12:54.358536 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:54.358742 7191 raft_consensus.cc:3037] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.361245 7191 raft_consensus.cc:2454] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:54.361402 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:54.361457 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:54.361687 6799 raft_consensus.cc:2454] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:54.361750 6996 leader_election.cc:302] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:54.362115 7909 raft_consensus.cc:2781] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:54.362517 7909 raft_consensus.cc:686] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:12:54.362812 7909 consensus_queue.cc:237] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:54.365625 6658 catalog_manager.cc:5484] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:54.396317 7903 raft_consensus.cc:482] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:54.396525 7903 raft_consensus.cc:504] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.397223 7903 leader_election.cc:288] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:54.397814 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24090e6169dd44f8b47c5cd772934710" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:54.398070 6930 raft_consensus.cc:2454] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:54.398540 7125 leader_election.cc:302] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:54.398871 7903 raft_consensus.cc:2781] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:54.399044 7903 raft_consensus.cc:482] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:54.399161 7903 raft_consensus.cc:3037] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.401664 7903 raft_consensus.cc:504] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.402478 7903 leader_election.cc:288] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:54.402969 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24090e6169dd44f8b47c5cd772934710" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:54.403172 6930 raft_consensus.cc:3037] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.405417 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24090e6169dd44f8b47c5cd772934710" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:54.405483 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24090e6169dd44f8b47c5cd772934710" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:54.405678 7060 raft_consensus.cc:2454] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:54.406031 6930 raft_consensus.cc:2454] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:54.406409 7125 leader_election.cc:302] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:12:54.406710 7903 raft_consensus.cc:2781] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:54.406859 7903 raft_consensus.cc:686] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:12:54.407169 7903 consensus_queue.cc:237] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:54.409986 6658 catalog_manager.cc:5484] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:54.421778 7909 raft_consensus.cc:482] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:54.422051 7909 raft_consensus.cc:504] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.422796 7909 leader_election.cc:288] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:54.423333 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca522e5b44fd44289b4b161f27f71d20" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:54.423451 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca522e5b44fd44289b4b161f27f71d20" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:54.423610 6799 raft_consensus.cc:2454] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:54.423718 6930 raft_consensus.cc:2454] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:54.424080 6995 leader_election.cc:302] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:12:54.424417 7909 raft_consensus.cc:2781] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:54.424539 7909 raft_consensus.cc:482] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:54.424661 7909 raft_consensus.cc:3037] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.426690 7909 raft_consensus.cc:504] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.427294 7909 leader_election.cc:288] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:54.427826 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca522e5b44fd44289b4b161f27f71d20" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:54.427837 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca522e5b44fd44289b4b161f27f71d20" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:54.427999 6799 raft_consensus.cc:3037] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.428025 6931 raft_consensus.cc:3037] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.430071 6799 raft_consensus.cc:2454] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:54.430105 6931 raft_consensus.cc:2454] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:54.430510 6995 leader_election.cc:302] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:54.430859 7909 raft_consensus.cc:2781] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:54.431048 7909 raft_consensus.cc:686] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:12:54.431335 7909 consensus_queue.cc:237] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:54.434307 6658 catalog_manager.cc:5484] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:54.792481 6799 raft_consensus.cc:3037] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.796569 7909 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:54.802738 8067 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:54.826920 8073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:54.827297 8073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:54.835193 8073 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240503 14:12:54.887528 7903 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:54.892333 7060 raft_consensus.cc:3037] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:54.897433 8025 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:54.967144 8067 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:54.976581 6931 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:12:54.977270 8067 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:55.012401 8102 mvcc.cc:205] Tried to move back new op lower bound from 7023597875098140672 to 7023597872873046016. Current Snapshot: MvccSnapshot[applied={T|T < 7023597875098140672}]
I20240503 14:12:55.084091 8075 mvcc.cc:205] Tried to move back new op lower bound from 7023597875098140672 to 7023597872873046016. Current Snapshot: MvccSnapshot[applied={T|T < 7023597875098140672 or (T in {7023597875098140672})}]
INSERT report
rows total: 2000
time total: 258.998 ms
time per row: 0.129499 ms
I20240503 14:12:55.241209 6659 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:55.251610 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 5015820c8edd42dc9e18010e60b4ca19 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.251729 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 5015820c8edd42dc9e18010e60b4ca19 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.252133 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5015820c8edd42dc9e18010e60b4ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.252152 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5015820c8edd42dc9e18010e60b4ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.256278 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 5015820c8edd42dc9e18010e60b4ca19 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.256757 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5015820c8edd42dc9e18010e60b4ca19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.256945 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 837b546bb5e141fb8143da58fbbb084a (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.257297 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837b546bb5e141fb8143da58fbbb084a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.258914 8136 tablet_bootstrap.cc:492] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:55.260061 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 534d2b4eec6547039bc06b9d7837ace4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.260478 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534d2b4eec6547039bc06b9d7837ace4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.260718 8136 tablet_bootstrap.cc:654] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.261456 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 837b546bb5e141fb8143da58fbbb084a (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.261814 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 534d2b4eec6547039bc06b9d7837ace4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.261932 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837b546bb5e141fb8143da58fbbb084a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.262198 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534d2b4eec6547039bc06b9d7837ace4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.264091 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 534d2b4eec6547039bc06b9d7837ace4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.264571 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 534d2b4eec6547039bc06b9d7837ace4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.267030 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 837b546bb5e141fb8143da58fbbb084a (DEFAULT_TABLE table=rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:55.267416 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 837b546bb5e141fb8143da58fbbb084a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:55.280475 8138 tablet_bootstrap.cc:492] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:55.282431 8138 tablet_bootstrap.cc:654] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.292349 8141 tablet_bootstrap.cc:492] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:55.295645 8141 tablet_bootstrap.cc:654] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.298509 8140 tablet_bootstrap.cc:492] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:55.300462 8138 tablet_bootstrap.cc:492] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:55.300664 8138 ts_tablet_manager.cc:1367] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.012s
I20240503 14:12:55.301139 8136 tablet_bootstrap.cc:492] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:55.301298 8136 ts_tablet_manager.cc:1367] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.042s user 0.014s sys 0.019s
I20240503 14:12:55.301540 8138 raft_consensus.cc:348] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.301787 8138 raft_consensus.cc:374] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.301875 8138 raft_consensus.cc:729] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.302052 8136 raft_consensus.cc:348] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.302235 8136 raft_consensus.cc:374] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.302340 8136 raft_consensus.cc:729] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.302599 8136 consensus_queue.cc:260] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.303256 8136 ts_tablet_manager.cc:1398] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:55.303426 8141 tablet_bootstrap.cc:492] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:55.303620 8141 ts_tablet_manager.cc:1367] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.002s
I20240503 14:12:55.304389 8141 raft_consensus.cc:348] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.304832 8136 tablet_bootstrap.cc:492] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:55.306624 8136 tablet_bootstrap.cc:654] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.304571 8141 raft_consensus.cc:374] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.307746 8141 raft_consensus.cc:729] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.308033 8141 consensus_queue.cc:260] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.308657 8141 ts_tablet_manager.cc:1398] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 14:12:55.308990 8141 tablet_bootstrap.cc:492] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:55.310485 8140 tablet_bootstrap.cc:654] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.310722 8141 tablet_bootstrap.cc:654] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.312222 8138 consensus_queue.cc:260] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.312559 8067 raft_consensus.cc:482] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:55.312670 8136 tablet_bootstrap.cc:492] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:55.312830 8136 ts_tablet_manager.cc:1367] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 14:12:55.312744 8067 raft_consensus.cc:504] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.312927 8138 ts_tablet_manager.cc:1398] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.012s user 0.001s sys 0.002s
I20240503 14:12:55.313264 8138 tablet_bootstrap.cc:492] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:55.313728 8067 leader_election.cc:288] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:55.313652 8136 raft_consensus.cc:348] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.313899 8136 raft_consensus.cc:374] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.313968 8136 raft_consensus.cc:729] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.314321 8136 consensus_queue.cc:260] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.315019 8136 ts_tablet_manager.cc:1398] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:55.315307 8138 tablet_bootstrap.cc:654] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.315326 8136 tablet_bootstrap.cc:492] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:55.316995 8136 tablet_bootstrap.cc:654] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.317886 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534d2b4eec6547039bc06b9d7837ace4" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:55.318109 7191 raft_consensus.cc:2454] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:55.319764 8141 tablet_bootstrap.cc:492] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:55.319975 8141 ts_tablet_manager.cc:1367] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20240503 14:12:55.321125 8141 raft_consensus.cc:348] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.321260 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534d2b4eec6547039bc06b9d7837ace4" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:55.321444 8141 raft_consensus.cc:374] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.321545 8141 raft_consensus.cc:729] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.321822 8141 consensus_queue.cc:260] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.322402 8141 ts_tablet_manager.cc:1398] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:55.322638 6996 leader_election.cc:302] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:55.322978 8123 raft_consensus.cc:2781] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:55.323089 8123 raft_consensus.cc:482] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240503 14:12:55.323210 6995 leader_election.cc:341] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:55.323207 8123 raft_consensus.cc:3037] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:55.324050 8136 tablet_bootstrap.cc:492] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:55.324213 8136 ts_tablet_manager.cc:1367] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:12:55.325122 8136 raft_consensus.cc:348] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.325332 8136 raft_consensus.cc:374] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.325435 8136 raft_consensus.cc:729] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.325774 8136 consensus_queue.cc:260] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.326416 8136 ts_tablet_manager.cc:1398] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:55.326560 8123 raft_consensus.cc:504] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.327322 8123 leader_election.cc:288] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:55.327854 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534d2b4eec6547039bc06b9d7837ace4" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:55.328070 6799 raft_consensus.cc:3037] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:55.328874 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "534d2b4eec6547039bc06b9d7837ace4" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:55.329149 7191 raft_consensus.cc:3037] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:55.330801 6799 raft_consensus.cc:2454] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:55.331331 6995 leader_election.cc:302] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:55.336942 7191 raft_consensus.cc:2454] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:55.337396 8123 raft_consensus.cc:2781] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:55.337589 8123 raft_consensus.cc:686] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:12:55.337848 8123 consensus_queue.cc:237] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:55.338034 8044 raft_consensus.cc:482] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:55.338284 8044 raft_consensus.cc:504] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.339155 8044 leader_election.cc:288] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:55.340291 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837b546bb5e141fb8143da58fbbb084a" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:55.340358 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837b546bb5e141fb8143da58fbbb084a" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:55.340581 7191 raft_consensus.cc:2454] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:55.340970 6735 leader_election.cc:302] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:55.341292 8044 raft_consensus.cc:2781] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 14:12:55.341202 6734 leader_election.cc:341] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:55.341403 8044 raft_consensus.cc:482] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:55.341498 8044 raft_consensus.cc:3037] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:55.345141 8044 raft_consensus.cc:504] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.345436 6659 catalog_manager.cc:5484] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:55.346513 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837b546bb5e141fb8143da58fbbb084a" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:55.346599 8044 leader_election.cc:288] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:55.346745 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837b546bb5e141fb8143da58fbbb084a" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:55.347055 7191 raft_consensus.cc:3037] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:12:55.347050 6734 leader_election.cc:341] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:12:55.350844 7191 raft_consensus.cc:2454] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:55.351639 6735 leader_election.cc:302] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters: 7186ed8510424fd597173eaf10c57b12
I20240503 14:12:55.353405 8140 tablet_bootstrap.cc:492] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:55.353610 8140 ts_tablet_manager.cc:1367] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20240503 14:12:55.353919 8044 raft_consensus.cc:2781] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:55.354352 8044 raft_consensus.cc:686] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:12:55.354700 8044 consensus_queue.cc:237] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.355113 8140 raft_consensus.cc:348] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.356423 8140 raft_consensus.cc:374] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.356542 8140 raft_consensus.cc:729] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.356777 8140 consensus_queue.cc:260] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.357537 8140 ts_tablet_manager.cc:1398] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:55.357822 8140 tablet_bootstrap.cc:492] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:55.358429 8138 tablet_bootstrap.cc:492] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:55.358688 8138 ts_tablet_manager.cc:1367] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.010s
I20240503 14:12:55.359556 8140 tablet_bootstrap.cc:654] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:55.360021 8138 raft_consensus.cc:348] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.360404 8138 raft_consensus.cc:374] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.360567 8138 raft_consensus.cc:729] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.360920 8138 consensus_queue.cc:260] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.361482 6659 catalog_manager.cc:5484] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:55.361780 8138 ts_tablet_manager.cc:1398] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:12:55.362625 8140 tablet_bootstrap.cc:492] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:55.362816 8140 ts_tablet_manager.cc:1367] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:12:55.364004 8140 raft_consensus.cc:348] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.364409 8140 raft_consensus.cc:374] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:55.364497 8140 raft_consensus.cc:729] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:55.364755 8140 consensus_queue.cc:260] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.365432 8140 ts_tablet_manager.cc:1398] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:12:55.377298 8015 raft_consensus.cc:482] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:55.377485 8015 raft_consensus.cc:504] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.378099 8015 leader_election.cc:288] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:12:55.378693 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837b546bb5e141fb8143da58fbbb084a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:55.378707 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "837b546bb5e141fb8143da58fbbb084a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:55.378963 7191 raft_consensus.cc:2379] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7186ed8510424fd597173eaf10c57b12 in current term 1: Already voted for candidate f1a669e28f014fe287932acf56bebc13 in this term.
I20240503 14:12:55.379527 6866 leader_election.cc:302] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12; no voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13
I20240503 14:12:55.379832 8015 raft_consensus.cc:3037] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:55.381820 8015 raft_consensus.cc:2726] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:12:55.557930 8044 raft_consensus.cc:482] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:55.558192 8044 raft_consensus.cc:504] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.558938 8044 leader_election.cc:288] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:55.559530 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5015820c8edd42dc9e18010e60b4ca19" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:55.559551 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5015820c8edd42dc9e18010e60b4ca19" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:55.559815 6931 raft_consensus.cc:2454] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:55.559834 7061 raft_consensus.cc:2454] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:55.560312 6734 leader_election.cc:302] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:55.560633 8044 raft_consensus.cc:2781] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:55.560739 8044 raft_consensus.cc:482] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:55.560850 8044 raft_consensus.cc:3037] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:55.562728 8044 raft_consensus.cc:504] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.563378 8044 leader_election.cc:288] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:55.563885 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5015820c8edd42dc9e18010e60b4ca19" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:55.563935 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5015820c8edd42dc9e18010e60b4ca19" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:55.564137 7061 raft_consensus.cc:3037] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:55.564152 6931 raft_consensus.cc:3037] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:55.567014 6931 raft_consensus.cc:2454] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:55.567013 7061 raft_consensus.cc:2454] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:55.567471 6734 leader_election.cc:302] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:55.567811 8044 raft_consensus.cc:2781] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:55.567981 8044 raft_consensus.cc:686] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:12:55.568241 8044 consensus_queue.cc:237] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:12:55.570870 6659 catalog_manager.cc:5484] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:55.826421 8044 consensus_queue.cc:1035] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:55.831673 8044 consensus_queue.cc:1035] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:55.834133 8123 consensus_queue.cc:1035] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:55.850690 8067 consensus_queue.cc:1035] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:55.978391 8153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:55.978758 8153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:55.987078 8153 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 14:12:56.048869 8044 consensus_queue.cc:1035] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:56.056273 8044 consensus_queue.cc:1035] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 236.209 ms
time per row: 0.118105 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:12:57.339058 6657 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:57.349109 7171 tablet_service.cc:1467] Processing CreateTablet for tablet c8848a40563c43468ac46f8c604cc578 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.349239 7040 tablet_service.cc:1467] Processing CreateTablet for tablet c8848a40563c43468ac46f8c604cc578 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.349627 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8848a40563c43468ac46f8c604cc578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.349689 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8848a40563c43468ac46f8c604cc578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.349761 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 669d722d3eb1475dba616ed72cb775ac (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.350098 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669d722d3eb1475dba616ed72cb775ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.350659 6911 tablet_service.cc:1467] Processing CreateTablet for tablet c8848a40563c43468ac46f8c604cc578 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.351048 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8848a40563c43468ac46f8c604cc578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.354995 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 669d722d3eb1475dba616ed72cb775ac (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.355398 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669d722d3eb1475dba616ed72cb775ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.355913 7039 tablet_service.cc:1467] Processing CreateTablet for tablet a90c8e47862945f1a873b17da14bb397 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.356240 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90c8e47862945f1a873b17da14bb397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.358085 6909 tablet_service.cc:1467] Processing CreateTablet for tablet a90c8e47862945f1a873b17da14bb397 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.358506 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90c8e47862945f1a873b17da14bb397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.359007 6778 tablet_service.cc:1467] Processing CreateTablet for tablet a90c8e47862945f1a873b17da14bb397 (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.359539 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90c8e47862945f1a873b17da14bb397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.358469 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 669d722d3eb1475dba616ed72cb775ac (DEFAULT_TABLE table=rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:57.361171 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669d722d3eb1475dba616ed72cb775ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:57.371191 8222 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:57.374540 8223 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:57.375424 8222 tablet_bootstrap.cc:654] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.377223 8223 tablet_bootstrap.cc:654] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.378801 8221 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:57.381044 8221 tablet_bootstrap.cc:654] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.382246 8226 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:57.383961 8226 tablet_bootstrap.cc:654] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.385354 8223 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:57.385524 8223 ts_tablet_manager.cc:1367] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:12:57.386747 8226 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:57.386894 8226 ts_tablet_manager.cc:1367] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:12:57.386770 8223 raft_consensus.cc:348] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.387205 8222 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:57.387313 8223 raft_consensus.cc:374] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.387367 8222 ts_tablet_manager.cc:1367] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240503 14:12:57.387512 8223 raft_consensus.cc:729] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.387687 8226 raft_consensus.cc:348] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.387889 8226 raft_consensus.cc:374] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.387987 8226 raft_consensus.cc:729] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.388168 8222 raft_consensus.cc:348] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.388109 8223 consensus_queue.cc:260] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.388254 8226 consensus_queue.cc:260] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.388370 8222 raft_consensus.cc:374] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.388451 8222 raft_consensus.cc:729] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.388669 8222 consensus_queue.cc:260] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.388828 8226 ts_tablet_manager.cc:1398] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:57.389348 8222 ts_tablet_manager.cc:1398] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:57.389633 8222 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:57.391983 8223 ts_tablet_manager.cc:1398] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:12:57.391999 8222 tablet_bootstrap.cc:654] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.392298 8223 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:57.393826 8223 tablet_bootstrap.cc:654] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.395457 8221 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:57.395653 8221 ts_tablet_manager.cc:1367] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20240503 14:12:57.396473 8221 raft_consensus.cc:348] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.397871 8222 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:57.397900 8223 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:57.398051 8223 ts_tablet_manager.cc:1367] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240503 14:12:57.398104 8222 ts_tablet_manager.cc:1367] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20240503 14:12:57.398897 8221 raft_consensus.cc:374] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.398859 8223 raft_consensus.cc:348] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.399087 8223 raft_consensus.cc:374] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.398934 8222 raft_consensus.cc:348] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.399129 8221 raft_consensus.cc:729] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.399258 8222 raft_consensus.cc:374] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.399250 8223 raft_consensus.cc:729] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.399449 8221 consensus_queue.cc:260] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.399528 8222 raft_consensus.cc:729] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.399547 8223 consensus_queue.cc:260] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.399921 8222 consensus_queue.cc:260] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.400159 8223 ts_tablet_manager.cc:1398] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:12:57.400861 8223 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:57.400849 8221 ts_tablet_manager.cc:1398] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:12:57.401109 8221 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:57.402056 8222 ts_tablet_manager.cc:1398] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:57.402375 8222 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:57.402535 8223 tablet_bootstrap.cc:654] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.402823 8221 tablet_bootstrap.cc:654] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.404124 8222 tablet_bootstrap.cc:654] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:57.410689 8222 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:57.410884 8221 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:57.411018 8222 ts_tablet_manager.cc:1367] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:12:57.411396 8221 ts_tablet_manager.cc:1367] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.001s
I20240503 14:12:57.412330 8221 raft_consensus.cc:348] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.412617 8221 raft_consensus.cc:374] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.412412 8222 raft_consensus.cc:348] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.412744 8221 raft_consensus.cc:729] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.412868 8222 raft_consensus.cc:374] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.413043 8222 raft_consensus.cc:729] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.413007 8221 consensus_queue.cc:260] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.413563 8221 ts_tablet_manager.cc:1398] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:57.413443 8222 consensus_queue.cc:260] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.415189 8222 ts_tablet_manager.cc:1398] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:57.416623 8223 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:57.416791 8223 ts_tablet_manager.cc:1367] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.010s
I20240503 14:12:57.417560 8223 raft_consensus.cc:348] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.417776 8223 raft_consensus.cc:374] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:57.417919 8223 raft_consensus.cc:729] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:57.418183 8223 consensus_queue.cc:260] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.419368 8223 ts_tablet_manager.cc:1398] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:57.444482 8196 raft_consensus.cc:482] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:57.444681 8196 raft_consensus.cc:504] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.445397 8196 leader_election.cc:288] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:57.445984 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:57.445979 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:57.446265 7190 raft_consensus.cc:2454] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:57.446349 6930 raft_consensus.cc:2454] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:57.446820 6995 leader_election.cc:302] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:12:57.447222 8196 raft_consensus.cc:2781] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:57.447407 8196 raft_consensus.cc:482] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:57.447510 8196 raft_consensus.cc:3037] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.449615 8196 raft_consensus.cc:504] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.450157 8196 leader_election.cc:288] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:57.450635 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:57.450678 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:57.450826 7190 raft_consensus.cc:3037] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.450883 6930 raft_consensus.cc:3037] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.452723 7190 raft_consensus.cc:2454] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:57.452939 6930 raft_consensus.cc:2454] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:57.453217 6996 leader_election.cc:302] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:57.453544 8196 raft_consensus.cc:2781] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:57.453927 8196 raft_consensus.cc:686] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:12:57.454193 8196 consensus_queue.cc:237] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:57.457479 6656 catalog_manager.cc:5484] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:57.467214 8196 raft_consensus.cc:482] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:57.467490 8196 raft_consensus.cc:504] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.468127 8196 leader_election.cc:288] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:57.468720 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90c8e47862945f1a873b17da14bb397" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:57.468745 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90c8e47862945f1a873b17da14bb397" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:57.468961 6930 raft_consensus.cc:2454] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:57.468961 6799 raft_consensus.cc:2454] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:57.469432 6995 leader_election.cc:302] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:57.469693 8196 raft_consensus.cc:2781] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:57.469794 8196 raft_consensus.cc:482] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:57.469899 8196 raft_consensus.cc:3037] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.497467 8196 raft_consensus.cc:504] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.498195 8196 leader_election.cc:288] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:57.498713 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90c8e47862945f1a873b17da14bb397" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:57.498811 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90c8e47862945f1a873b17da14bb397" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:57.498977 6799 raft_consensus.cc:3037] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.499056 6930 raft_consensus.cc:3037] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.501011 6799 raft_consensus.cc:2454] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:57.501211 6930 raft_consensus.cc:2454] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:57.501528 6995 leader_election.cc:302] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:57.501900 8196 raft_consensus.cc:2781] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:57.502085 8196 raft_consensus.cc:686] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:12:57.502352 8196 consensus_queue.cc:237] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.505080 6656 catalog_manager.cc:5484] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:57.510965 8196 raft_consensus.cc:482] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:57.511124 8196 raft_consensus.cc:504] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.511901 8196 leader_election.cc:288] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:57.512327 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:57.512404 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:57.512581 7190 raft_consensus.cc:2454] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:57.512625 6929 raft_consensus.cc:2454] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:12:57.513069 6996 leader_election.cc:302] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:57.513481 8196 raft_consensus.cc:2781] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:57.513597 8196 raft_consensus.cc:482] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:57.513696 8196 raft_consensus.cc:3037] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.515692 8196 raft_consensus.cc:504] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.516218 8196 leader_election.cc:288] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:57.516750 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:57.516796 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:57.516938 6929 raft_consensus.cc:3037] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.517009 7190 raft_consensus.cc:3037] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:57.518940 6929 raft_consensus.cc:2454] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:57.518975 7190 raft_consensus.cc:2454] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:12:57.519418 6996 leader_election.cc:302] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:57.519778 8196 raft_consensus.cc:2781] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:57.520005 8196 raft_consensus.cc:686] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:12:57.520280 8196 consensus_queue.cc:237] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:57.523562 6656 catalog_manager.cc:5484] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:57.905952 8196 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:57.917412 8234 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:57.930071 8234 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:57.944734 8238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:57.945212 8238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:57.954495 8238 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:57.965878 8234 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:57.970984 8196 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:57.975180 8196 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_5'
W20240503 14:12:58.206368 6995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.2:45285, user_credentials={real_user=slave}} blocked reactor thread for 66102.6us
INSERT report
rows total: 2000
time total: 228.18 ms
time per row: 0.11409 ms
I20240503 14:12:58.369943 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:58.379213 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 2448fc37870d4d3b9bf267da36780f94 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.379618 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 2448fc37870d4d3b9bf267da36780f94 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.379804 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448fc37870d4d3b9bf267da36780f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.380038 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448fc37870d4d3b9bf267da36780f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.380015 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 2448fc37870d4d3b9bf267da36780f94 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.380390 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2448fc37870d4d3b9bf267da36780f94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.381731 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 310a9ef153524468abb5a00808d186c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.382189 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310a9ef153524468abb5a00808d186c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.383394 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 09400f720f464ef8b8bec4549826632e (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.383874 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09400f720f464ef8b8bec4549826632e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.386587 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 310a9ef153524468abb5a00808d186c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.386960 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310a9ef153524468abb5a00808d186c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.388070 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 09400f720f464ef8b8bec4549826632e (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.388535 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09400f720f464ef8b8bec4549826632e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.391211 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 09400f720f464ef8b8bec4549826632e (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.391777 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09400f720f464ef8b8bec4549826632e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.396070 8301 tablet_bootstrap.cc:492] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:58.396263 7169 tablet_service.cc:1467] Processing CreateTablet for tablet 310a9ef153524468abb5a00808d186c6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:58.396862 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310a9ef153524468abb5a00808d186c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:58.399879 8302 tablet_bootstrap.cc:492] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:58.400028 8301 tablet_bootstrap.cc:654] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.404302 8302 tablet_bootstrap.cc:654] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.413573 8299 tablet_bootstrap.cc:492] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:58.414417 8301 tablet_bootstrap.cc:492] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:58.414686 8301 ts_tablet_manager.cc:1367] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.006s
I20240503 14:12:58.415820 8299 tablet_bootstrap.cc:654] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.418668 8301 raft_consensus.cc:348] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.419059 8300 tablet_bootstrap.cc:492] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:58.419176 8301 raft_consensus.cc:374] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.419338 8301 raft_consensus.cc:729] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.419847 8302 tablet_bootstrap.cc:492] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:58.419742 8301 consensus_queue.cc:260] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.420166 8302 ts_tablet_manager.cc:1367] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240503 14:12:58.421485 8300 tablet_bootstrap.cc:654] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.421880 8301 ts_tablet_manager.cc:1398] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:12:58.424253 8301 tablet_bootstrap.cc:492] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:58.421124 8302 raft_consensus.cc:348] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.425854 8302 raft_consensus.cc:374] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.425998 8302 raft_consensus.cc:729] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.428370 8301 tablet_bootstrap.cc:654] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.428872 8302 consensus_queue.cc:260] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.433562 8302 ts_tablet_manager.cc:1398] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:12:58.434384 8301 tablet_bootstrap.cc:492] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:58.434412 8299 tablet_bootstrap.cc:492] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:58.434648 8301 ts_tablet_manager.cc:1367] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20240503 14:12:58.434796 8299 ts_tablet_manager.cc:1367] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.015s
I20240503 14:12:58.435647 8301 raft_consensus.cc:348] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.435902 8301 raft_consensus.cc:374] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.435987 8301 raft_consensus.cc:729] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.436311 8301 consensus_queue.cc:260] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.436924 8301 ts_tablet_manager.cc:1398] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:58.436362 8299 raft_consensus.cc:348] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.437220 8299 raft_consensus.cc:374] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.437644 8301 tablet_bootstrap.cc:492] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:58.437924 8299 raft_consensus.cc:729] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.438271 8299 consensus_queue.cc:260] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.438977 8299 ts_tablet_manager.cc:1398] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:12:58.439503 8301 tablet_bootstrap.cc:654] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.439742 8300 tablet_bootstrap.cc:492] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:58.439904 8300 ts_tablet_manager.cc:1367] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.016s
I20240503 14:12:58.441227 8300 raft_consensus.cc:348] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.441465 8300 raft_consensus.cc:374] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.441569 8300 raft_consensus.cc:729] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.441818 8300 consensus_queue.cc:260] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.443344 8299 tablet_bootstrap.cc:492] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:58.443711 8300 ts_tablet_manager.cc:1398] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:12:58.444006 8300 tablet_bootstrap.cc:492] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:58.445531 8299 tablet_bootstrap.cc:654] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.446547 8300 tablet_bootstrap.cc:654] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.448707 8300 tablet_bootstrap.cc:492] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:58.448868 8300 ts_tablet_manager.cc:1367] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:12:58.449453 8301 tablet_bootstrap.cc:492] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:58.449529 8300 raft_consensus.cc:348] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.449683 8301 ts_tablet_manager.cc:1367] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.002s
I20240503 14:12:58.449739 8300 raft_consensus.cc:374] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.449828 8300 raft_consensus.cc:729] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.450073 8300 consensus_queue.cc:260] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.450708 8300 ts_tablet_manager.cc:1398] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:58.450810 8301 raft_consensus.cc:348] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.451042 8301 raft_consensus.cc:374] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.451156 8301 raft_consensus.cc:729] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.451468 8301 consensus_queue.cc:260] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.452440 8299 tablet_bootstrap.cc:492] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:58.452627 8301 ts_tablet_manager.cc:1398] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:12:58.452682 8299 ts_tablet_manager.cc:1367] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20240503 14:12:58.453593 8299 raft_consensus.cc:348] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.453835 8299 raft_consensus.cc:374] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.453918 8299 raft_consensus.cc:729] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.454174 8299 consensus_queue.cc:260] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.455296 8299 ts_tablet_manager.cc:1398] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:12:58.455621 8299 tablet_bootstrap.cc:492] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:58.456887 8299 tablet_bootstrap.cc:654] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:58.459499 8299 tablet_bootstrap.cc:492] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:58.459667 8299 ts_tablet_manager.cc:1367] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:12:58.460287 8299 raft_consensus.cc:348] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.460495 8299 raft_consensus.cc:374] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:58.460597 8299 raft_consensus.cc:729] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:58.460804 8299 consensus_queue.cc:260] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.461274 8299 ts_tablet_manager.cc:1398] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:12:58.482105 7903 raft_consensus.cc:482] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:58.482283 7903 raft_consensus.cc:504] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.482965 7903 leader_election.cc:288] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:58.483456 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09400f720f464ef8b8bec4549826632e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:58.483726 6800 raft_consensus.cc:2454] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:58.483562 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09400f720f464ef8b8bec4549826632e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:58.483911 7060 raft_consensus.cc:2454] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:58.484213 7125 leader_election.cc:302] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:58.484519 7903 raft_consensus.cc:2781] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:58.484651 7903 raft_consensus.cc:482] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:58.484746 7903 raft_consensus.cc:3037] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.486733 7903 raft_consensus.cc:504] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.487394 7903 leader_election.cc:288] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:58.487876 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09400f720f464ef8b8bec4549826632e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:58.487947 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09400f720f464ef8b8bec4549826632e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:58.488094 6800 raft_consensus.cc:3037] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.488144 7060 raft_consensus.cc:3037] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.489954 6800 raft_consensus.cc:2454] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:58.490006 7060 raft_consensus.cc:2454] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:58.490351 7125 leader_election.cc:302] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:58.490634 7903 raft_consensus.cc:2781] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:58.491003 7903 raft_consensus.cc:686] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:12:58.491278 7903 consensus_queue.cc:237] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:12:58.494451 6656 catalog_manager.cc:5484] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:58.559058 8149 raft_consensus.cc:482] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:58.559273 8149 raft_consensus.cc:504] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.560030 8149 leader_election.cc:288] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:58.560658 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2448fc37870d4d3b9bf267da36780f94" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:58.560678 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2448fc37870d4d3b9bf267da36780f94" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:58.560895 7189 raft_consensus.cc:2454] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:58.560899 6931 raft_consensus.cc:2454] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:58.561373 6735 leader_election.cc:302] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:58.561676 8149 raft_consensus.cc:2781] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:58.561833 8149 raft_consensus.cc:482] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:58.561921 8149 raft_consensus.cc:3037] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.564213 8149 raft_consensus.cc:504] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.564940 8149 leader_election.cc:288] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:58.565505 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2448fc37870d4d3b9bf267da36780f94" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:58.565536 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2448fc37870d4d3b9bf267da36780f94" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:58.565716 7189 raft_consensus.cc:3037] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.565748 6931 raft_consensus.cc:3037] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.567852 6931 raft_consensus.cc:2454] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:58.567965 7189 raft_consensus.cc:2454] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:58.568460 6734 leader_election.cc:302] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:58.568817 8149 raft_consensus.cc:2781] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:58.569029 8149 raft_consensus.cc:686] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:12:58.569331 8149 consensus_queue.cc:237] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.572463 6656 catalog_manager.cc:5484] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:12:58.621559 8248 raft_consensus.cc:482] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:58.621779 8248 raft_consensus.cc:504] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.622646 8248 leader_election.cc:288] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:58.623013 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310a9ef153524468abb5a00808d186c6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:58.623147 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310a9ef153524468abb5a00808d186c6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:12:58.623220 6800 raft_consensus.cc:2454] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:12:58.623397 7189 raft_consensus.cc:2454] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:12:58.623644 6865 leader_election.cc:302] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:58.623901 8248 raft_consensus.cc:2781] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:58.623999 8248 raft_consensus.cc:482] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:58.624083 8248 raft_consensus.cc:3037] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.625936 8248 raft_consensus.cc:504] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.626498 8248 leader_election.cc:288] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:12:58.627060 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310a9ef153524468abb5a00808d186c6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:58.627049 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "310a9ef153524468abb5a00808d186c6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:12:58.627267 6800 raft_consensus.cc:3037] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.627293 7189 raft_consensus.cc:3037] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:58.630198 6800 raft_consensus.cc:2454] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:12:58.630203 7189 raft_consensus.cc:2454] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:12:58.630718 6866 leader_election.cc:302] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:58.631070 8248 raft_consensus.cc:2781] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:58.631244 8248 raft_consensus.cc:686] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:12:58.632001 8248 consensus_queue.cc:237] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:58.635577 6656 catalog_manager.cc:5484] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:58.869845 7903 consensus_queue.cc:1035] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:58.891053 7903 consensus_queue.cc:1035] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:12:59.063189 8317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:12:59.063577 8317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:12:59.071415 8317 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:12:59.098049 8149 consensus_queue.cc:1035] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:59.103539 8149 consensus_queue.cc:1035] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240503 14:12:59.169162 8248 consensus_queue.cc:1035] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:12:59.178954 8351 consensus_queue.cc:1035] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 275.742 ms
time per row: 0.137871 ms
I20240503 14:12:59.505932 6657 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:12:59.515887 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 34773b2ab9194e70b3e49357a2da4d8c (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.516402 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34773b2ab9194e70b3e49357a2da4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.516628 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 34773b2ab9194e70b3e49357a2da4d8c (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.517089 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34773b2ab9194e70b3e49357a2da4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.517055 7171 tablet_service.cc:1467] Processing CreateTablet for tablet c95eaefadbff4511b610ab85cdfeb909 (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.517469 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95eaefadbff4511b610ab85cdfeb909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.519901 7039 tablet_service.cc:1467] Processing CreateTablet for tablet c95eaefadbff4511b610ab85cdfeb909 (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.520103 6779 tablet_service.cc:1467] Processing CreateTablet for tablet c95eaefadbff4511b610ab85cdfeb909 (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.520320 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95eaefadbff4511b610ab85cdfeb909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.520480 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95eaefadbff4511b610ab85cdfeb909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.524230 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 34773b2ab9194e70b3e49357a2da4d8c (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.524680 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34773b2ab9194e70b3e49357a2da4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.525048 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3ec4f0833d4d489f774757312efb32 (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.525465 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3ec4f0833d4d489f774757312efb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.529757 8379 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:59.530781 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3ec4f0833d4d489f774757312efb32 (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.531231 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3ec4f0833d4d489f774757312efb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.532104 7169 tablet_service.cc:1467] Processing CreateTablet for tablet 1d3ec4f0833d4d489f774757312efb32 (DEFAULT_TABLE table=rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:12:59.532630 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3ec4f0833d4d489f774757312efb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:12:59.534796 8379 tablet_bootstrap.cc:654] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.542037 8380 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:59.542500 8379 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:59.543357 8379 ts_tablet_manager.cc:1367] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 14:12:59.544101 8380 tablet_bootstrap.cc:654] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.544417 8379 raft_consensus.cc:348] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.544781 8379 raft_consensus.cc:374] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.544934 8379 raft_consensus.cc:729] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.545315 8379 consensus_queue.cc:260] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.547660 8379 ts_tablet_manager.cc:1398] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:59.548151 8379 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:12:59.551937 8379 tablet_bootstrap.cc:654] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.553294 8383 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:59.555886 8383 tablet_bootstrap.cc:654] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.556728 8382 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:59.558329 8380 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:59.558641 8380 ts_tablet_manager.cc:1367] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.008s
I20240503 14:12:59.558903 8379 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:12:59.559167 8379 ts_tablet_manager.cc:1367] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240503 14:12:59.559650 8380 raft_consensus.cc:348] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.559919 8380 raft_consensus.cc:374] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.560067 8380 raft_consensus.cc:729] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.560312 8383 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:59.560302 8379 raft_consensus.cc:348] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.560346 8380 consensus_queue.cc:260] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.560506 8383 ts_tablet_manager.cc:1367] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20240503 14:12:59.560536 8379 raft_consensus.cc:374] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.560612 8379 raft_consensus.cc:729] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.561719 8382 tablet_bootstrap.cc:654] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.561468 8383 raft_consensus.cc:348] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.561923 8383 raft_consensus.cc:374] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.562062 8383 raft_consensus.cc:729] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.562240 8380 ts_tablet_manager.cc:1398] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:12:59.562368 8383 consensus_queue.cc:260] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.562561 8380 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:12:59.562881 8379 consensus_queue.cc:260] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.563058 8383 ts_tablet_manager.cc:1398] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:59.563350 8383 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:59.564441 8379 ts_tablet_manager.cc:1398] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:12:59.564862 8380 tablet_bootstrap.cc:654] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.567829 8383 tablet_bootstrap.cc:654] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.573410 8382 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:59.573747 8382 ts_tablet_manager.cc:1367] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.006s
I20240503 14:12:59.574754 8382 raft_consensus.cc:348] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.574981 8382 raft_consensus.cc:374] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.575038 8383 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:59.575143 8382 raft_consensus.cc:729] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.575352 8383 ts_tablet_manager.cc:1367] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.005s
I20240503 14:12:59.575543 8380 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:12:59.575416 8382 consensus_queue.cc:260] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.575763 8380 ts_tablet_manager.cc:1367] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240503 14:12:59.576184 8382 ts_tablet_manager.cc:1398] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:12:59.576539 8382 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:12:59.576642 8380 raft_consensus.cc:348] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.576866 8380 raft_consensus.cc:374] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.576968 8380 raft_consensus.cc:729] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.577239 8380 consensus_queue.cc:260] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.576287 8383 raft_consensus.cc:348] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.577488 8383 raft_consensus.cc:374] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.577847 8380 ts_tablet_manager.cc:1398] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:59.577834 8383 raft_consensus.cc:729] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.578131 8383 consensus_queue.cc:260] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.579249 8383 ts_tablet_manager.cc:1398] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:12:59.579556 8383 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:12:59.581321 8382 tablet_bootstrap.cc:654] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.581629 8383 tablet_bootstrap.cc:654] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:12:59.586014 8382 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:12:59.586222 8382 ts_tablet_manager.cc:1367] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240503 14:12:59.586964 8382 raft_consensus.cc:348] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.587146 8382 raft_consensus.cc:374] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.587225 8382 raft_consensus.cc:729] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.587500 8382 consensus_queue.cc:260] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.588223 8382 ts_tablet_manager.cc:1398] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:12:59.591029 8383 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:12:59.591224 8383 ts_tablet_manager.cc:1367] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240503 14:12:59.591982 8383 raft_consensus.cc:348] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.592185 8383 raft_consensus.cc:374] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:12:59.592279 8383 raft_consensus.cc:729] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:12:59.592514 8383 consensus_queue.cc:260] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.593140 8383 ts_tablet_manager.cc:1398] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:12:59.603474 8312 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:59.603698 8312 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.604499 8312 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:59.604975 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:59.605017 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:59.605234 6931 raft_consensus.cc:2454] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:59.605234 7061 raft_consensus.cc:2454] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:59.605692 7125 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:59.605960 8312 raft_consensus.cc:2781] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:59.606070 8312 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:59.606161 8312 raft_consensus.cc:3037] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.608145 8312 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.608691 8312 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:59.609119 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:59.609179 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:59.609316 6931 raft_consensus.cc:3037] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.609351 7061 raft_consensus.cc:3037] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.611178 7061 raft_consensus.cc:2454] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:59.611284 6931 raft_consensus.cc:2454] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:59.611639 7127 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:12:59.611965 8312 raft_consensus.cc:2781] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:59.612111 8312 raft_consensus.cc:686] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:12:59.612413 8312 consensus_queue.cc:237] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.615669 6659 catalog_manager.cc:5484] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:59.668241 8312 raft_consensus.cc:482] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:59.668470 8312 raft_consensus.cc:504] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.669307 8312 leader_election.cc:288] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:59.669637 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95eaefadbff4511b610ab85cdfeb909" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:12:59.669744 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95eaefadbff4511b610ab85cdfeb909" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:59.669864 6799 raft_consensus.cc:2454] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:59.669994 7061 raft_consensus.cc:2454] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:12:59.670327 7125 leader_election.cc:302] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:59.670727 8312 raft_consensus.cc:2781] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:59.670820 8312 raft_consensus.cc:482] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:59.670948 8312 raft_consensus.cc:3037] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.672999 8312 raft_consensus.cc:504] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.673592 8312 leader_election.cc:288] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:12:59.674132 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95eaefadbff4511b610ab85cdfeb909" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:59.674149 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95eaefadbff4511b610ab85cdfeb909" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:12:59.674346 6799 raft_consensus.cc:3037] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.674371 7061 raft_consensus.cc:3037] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.677295 7061 raft_consensus.cc:2454] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:59.677309 6799 raft_consensus.cc:2454] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:12:59.677803 7125 leader_election.cc:302] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:59.678130 8312 raft_consensus.cc:2781] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:59.678294 8312 raft_consensus.cc:686] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:12:59.678579 8312 consensus_queue.cc:237] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:12:59.681645 6659 catalog_manager.cc:5484] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:12:59.810621 8335 raft_consensus.cc:482] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:12:59.810808 8335 raft_consensus.cc:504] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.811563 8335 leader_election.cc:288] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:59.812202 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34773b2ab9194e70b3e49357a2da4d8c" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:12:59.812220 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34773b2ab9194e70b3e49357a2da4d8c" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:12:59.812433 7061 raft_consensus.cc:2454] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:59.812492 6931 raft_consensus.cc:2454] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:12:59.812897 6736 leader_election.cc:302] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:59.813210 8335 raft_consensus.cc:2781] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:12:59.813338 8335 raft_consensus.cc:482] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:12:59.813542 8335 raft_consensus.cc:3037] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.816097 8335 raft_consensus.cc:504] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.816727 8335 leader_election.cc:288] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:12:59.817219 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34773b2ab9194e70b3e49357a2da4d8c" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:12:59.817258 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34773b2ab9194e70b3e49357a2da4d8c" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:12:59.817404 7061 raft_consensus.cc:3037] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.817533 6931 raft_consensus.cc:3037] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:12:59.819398 7061 raft_consensus.cc:2454] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:59.819664 6931 raft_consensus.cc:2454] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:12:59.819969 6736 leader_election.cc:302] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:12:59.820309 8335 raft_consensus.cc:2781] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:12:59.820469 8335 raft_consensus.cc:686] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:12:59.820755 8335 consensus_queue.cc:237] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:12:59.823436 6659 catalog_manager.cc:5484] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:00.069765 8312 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.086083 8399 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.141908 8312 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.148855 7903 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:00.199381 8395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:00.199805 8395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:00.207674 8395 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240503 14:13:00.326710 8335 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:00.344060 6931 raft_consensus.cc:1261] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:00.344708 8432 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 228.434 ms
time per row: 0.114217 ms
I20240503 14:13:00.603986 6659 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:00.612785 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 46d94c6f01544219a5602b3f06cb35b7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.612922 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 46d94c6f01544219a5602b3f06cb35b7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.613126 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 46d94c6f01544219a5602b3f06cb35b7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.613312 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d94c6f01544219a5602b3f06cb35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.613312 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d94c6f01544219a5602b3f06cb35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.613615 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46d94c6f01544219a5602b3f06cb35b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.614732 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 8d111a5f08914420a7a429987c9699e9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.615173 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d111a5f08914420a7a429987c9699e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.615341 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 8d111a5f08914420a7a429987c9699e9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.616730 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 8d111a5f08914420a7a429987c9699e9 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.617146 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d111a5f08914420a7a429987c9699e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.619107 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8f4639ce314b34b88f67315f3e4fa7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.619488 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8f4639ce314b34b88f67315f3e4fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.619954 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8f4639ce314b34b88f67315f3e4fa7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.620313 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8f4639ce314b34b88f67315f3e4fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.624308 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d111a5f08914420a7a429987c9699e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.629326 7169 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8f4639ce314b34b88f67315f3e4fa7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:00.629909 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8f4639ce314b34b88f67315f3e4fa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:00.631152 8456 tablet_bootstrap.cc:492] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:00.631999 8458 tablet_bootstrap.cc:492] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:00.632998 8456 tablet_bootstrap.cc:654] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.633805 8458 tablet_bootstrap.cc:654] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.638573 8456 tablet_bootstrap.cc:492] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:00.639940 8456 ts_tablet_manager.cc:1367] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:00.640312 8458 tablet_bootstrap.cc:492] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:00.640558 8458 ts_tablet_manager.cc:1367] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 14:13:00.641063 8456 raft_consensus.cc:348] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.641283 8456 raft_consensus.cc:374] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.641372 8456 raft_consensus.cc:729] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.641578 8458 raft_consensus.cc:348] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.641610 8456 consensus_queue.cc:260] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.641875 8458 raft_consensus.cc:374] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.642007 8458 raft_consensus.cc:729] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.642210 8456 ts_tablet_manager.cc:1398] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:00.642329 8458 consensus_queue.cc:260] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.646118 8458 ts_tablet_manager.cc:1398] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:00.647603 8458 tablet_bootstrap.cc:492] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:00.650972 8458 tablet_bootstrap.cc:654] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.656320 8456 tablet_bootstrap.cc:492] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:00.656814 8457 tablet_bootstrap.cc:492] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:00.658519 8456 tablet_bootstrap.cc:654] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.659263 8457 tablet_bootstrap.cc:654] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.659763 8458 tablet_bootstrap.cc:492] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:00.660048 8458 ts_tablet_manager.cc:1367] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20240503 14:13:00.661123 8458 raft_consensus.cc:348] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.661504 8458 raft_consensus.cc:374] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.661643 8458 raft_consensus.cc:729] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.662036 8458 consensus_queue.cc:260] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.664016 8458 ts_tablet_manager.cc:1398] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:00.666617 8456 tablet_bootstrap.cc:492] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:00.666893 8456 ts_tablet_manager.cc:1367] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 14:13:00.667790 8456 raft_consensus.cc:348] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.668090 8456 raft_consensus.cc:374] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.668184 8462 tablet_bootstrap.cc:492] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:00.668215 8456 raft_consensus.cc:729] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.668546 8456 consensus_queue.cc:260] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.669333 8456 ts_tablet_manager.cc:1398] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:00.670183 8462 tablet_bootstrap.cc:654] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.672802 8457 tablet_bootstrap.cc:492] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:00.672896 8462 tablet_bootstrap.cc:492] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:00.672966 8457 ts_tablet_manager.cc:1367] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20240503 14:13:00.673051 8462 ts_tablet_manager.cc:1367] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 14:13:00.673760 8457 raft_consensus.cc:348] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.673991 8457 raft_consensus.cc:374] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.674086 8457 raft_consensus.cc:729] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.674211 8462 raft_consensus.cc:348] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.674392 8462 raft_consensus.cc:374] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.674329 8457 consensus_queue.cc:260] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.674486 8462 raft_consensus.cc:729] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.674705 8462 consensus_queue.cc:260] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.674849 8457 ts_tablet_manager.cc:1398] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 14:13:00.675148 8457 tablet_bootstrap.cc:492] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:00.675254 8462 ts_tablet_manager.cc:1398] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:00.675529 8462 tablet_bootstrap.cc:492] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:00.676905 8457 tablet_bootstrap.cc:654] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.680408 8462 tablet_bootstrap.cc:654] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.682574 8457 tablet_bootstrap.cc:492] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:00.682739 8457 ts_tablet_manager.cc:1367] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:13:00.683311 8457 raft_consensus.cc:348] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.683496 8457 raft_consensus.cc:374] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.683570 8457 raft_consensus.cc:729] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.683852 8457 consensus_queue.cc:260] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.684001 8462 tablet_bootstrap.cc:492] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:00.684151 8462 ts_tablet_manager.cc:1367] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:13:00.684439 8457 ts_tablet_manager.cc:1398] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:00.684830 8457 tablet_bootstrap.cc:492] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:00.685788 8462 raft_consensus.cc:348] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.686106 8462 raft_consensus.cc:374] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.686231 8462 raft_consensus.cc:729] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.686622 8457 tablet_bootstrap.cc:654] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:00.686600 8462 consensus_queue.cc:260] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.688971 8462 ts_tablet_manager.cc:1398] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:00.692728 8457 tablet_bootstrap.cc:492] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:00.692874 8457 ts_tablet_manager.cc:1367] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20240503 14:13:00.693478 8457 raft_consensus.cc:348] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.693639 8457 raft_consensus.cc:374] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:00.693733 8457 raft_consensus.cc:729] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:00.693953 8457 consensus_queue.cc:260] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.694453 8457 ts_tablet_manager.cc:1398] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:00.764685 8335 raft_consensus.cc:482] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:00.764894 8335 raft_consensus.cc:504] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.765622 8335 leader_election.cc:288] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:00.766144 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d94c6f01544219a5602b3f06cb35b7" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:00.766152 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d94c6f01544219a5602b3f06cb35b7" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:00.766387 7189 raft_consensus.cc:2454] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:00.766412 6931 raft_consensus.cc:2454] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:00.766957 6735 leader_election.cc:302] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:00.767263 8335 raft_consensus.cc:2781] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:00.767393 8335 raft_consensus.cc:482] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:00.767458 8335 raft_consensus.cc:3037] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.769397 8335 raft_consensus.cc:504] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.769973 8335 leader_election.cc:288] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:00.770541 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d94c6f01544219a5602b3f06cb35b7" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:00.770579 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46d94c6f01544219a5602b3f06cb35b7" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:00.770771 7189 raft_consensus.cc:3037] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.770797 6931 raft_consensus.cc:3037] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.773641 6931 raft_consensus.cc:2454] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:00.773667 7189 raft_consensus.cc:2454] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:00.774128 6735 leader_election.cc:302] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:00.774434 8335 raft_consensus.cc:2781] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:00.774583 8335 raft_consensus.cc:686] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:00.774899 8335 consensus_queue.cc:237] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.777853 6652 catalog_manager.cc:5484] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:00.832146 8248 raft_consensus.cc:482] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:00.832381 8248 raft_consensus.cc:504] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.833153 8248 leader_election.cc:288] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:00.833768 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d111a5f08914420a7a429987c9699e9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:00.833742 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d111a5f08914420a7a429987c9699e9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:00.834009 7059 raft_consensus.cc:2454] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:00.834028 6799 raft_consensus.cc:2454] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:00.834502 6865 leader_election.cc:302] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:00.834775 8248 raft_consensus.cc:2781] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:00.834901 8248 raft_consensus.cc:482] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:00.834995 8248 raft_consensus.cc:3037] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.836987 8248 raft_consensus.cc:504] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.837553 8248 leader_election.cc:288] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:00.838025 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d111a5f08914420a7a429987c9699e9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:00.838035 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d111a5f08914420a7a429987c9699e9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:00.838238 7059 raft_consensus.cc:3037] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.838258 6799 raft_consensus.cc:3037] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.840618 6799 raft_consensus.cc:2454] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:00.840638 7059 raft_consensus.cc:2454] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:00.841055 6867 leader_election.cc:302] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:00.841552 8248 raft_consensus.cc:2781] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:00.841702 8248 raft_consensus.cc:686] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:00.842002 8248 consensus_queue.cc:237] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:00.844993 6652 catalog_manager.cc:5484] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:00.861912 8234 raft_consensus.cc:482] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:00.862125 8234 raft_consensus.cc:504] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.862931 8234 leader_election.cc:288] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:00.863379 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8f4639ce314b34b88f67315f3e4fa7" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:00.863399 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8f4639ce314b34b88f67315f3e4fa7" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:00.863623 6799 raft_consensus.cc:2454] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:00.863651 7189 raft_consensus.cc:2454] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:00.864004 6995 leader_election.cc:302] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:00.864260 8234 raft_consensus.cc:2781] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:00.864337 8234 raft_consensus.cc:482] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:00.864429 8234 raft_consensus.cc:3037] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.866475 8234 raft_consensus.cc:504] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.867339 8234 leader_election.cc:288] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:00.867645 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8f4639ce314b34b88f67315f3e4fa7" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:00.867770 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8f4639ce314b34b88f67315f3e4fa7" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:00.867889 7189 raft_consensus.cc:3037] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.867981 6799 raft_consensus.cc:3037] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:00.870738 7189 raft_consensus.cc:2454] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:00.870744 6799 raft_consensus.cc:2454] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:00.871205 6995 leader_election.cc:302] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:00.871513 8234 raft_consensus.cc:2781] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:00.871719 8234 raft_consensus.cc:686] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:00.872017 8234 consensus_queue.cc:237] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:00.875254 6652 catalog_manager.cc:5484] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:01.238370 8335 consensus_queue.cc:1035] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:01.242015 8428 consensus_queue.cc:1035] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:01.297818 8234 consensus_queue.cc:1035] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:01.318617 8443 consensus_queue.cc:1035] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:01.331146 8473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:01.331540 8473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:01.339880 8473 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:01.371323 8248 consensus_queue.cc:1035] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:01.376992 8248 consensus_queue.cc:1035] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 315.606 ms
time per row: 0.157803 ms
I20240503 14:13:01.820909 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:01.830793 7169 tablet_service.cc:1467] Processing CreateTablet for tablet 1c727d4554464442a6a81a8ad4996c84 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.831161 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 1c727d4554464442a6a81a8ad4996c84 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.831328 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c727d4554464442a6a81a8ad4996c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.831405 6910 tablet_service.cc:1467] Processing CreateTablet for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.831674 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c727d4554464442a6a81a8ad4996c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.831918 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.831984 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 1c727d4554464442a6a81a8ad4996c84 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.832396 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c727d4554464442a6a81a8ad4996c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.836078 7171 tablet_service.cc:1467] Processing CreateTablet for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.836599 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.840224 7040 tablet_service.cc:1467] Processing CreateTablet for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.840893 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.848536 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 343520037eac42c4ac76fa1cc5e26fc3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.848835 8539 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:01.849079 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343520037eac42c4ac76fa1cc5e26fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.849673 6909 tablet_service.cc:1467] Processing CreateTablet for tablet 343520037eac42c4ac76fa1cc5e26fc3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.850072 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343520037eac42c4ac76fa1cc5e26fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.850898 8539 tablet_bootstrap.cc:654] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.854972 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 343520037eac42c4ac76fa1cc5e26fc3 (DEFAULT_TABLE table=rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:01.855634 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343520037eac42c4ac76fa1cc5e26fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:01.870155 8542 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:01.874698 8542 tablet_bootstrap.cc:654] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.875609 8540 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:01.876848 8538 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:01.877888 8540 tablet_bootstrap.cc:654] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.879555 8538 tablet_bootstrap.cc:654] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.881078 8539 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:01.881280 8539 ts_tablet_manager.cc:1367] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.020s
I20240503 14:13:01.882458 8539 raft_consensus.cc:348] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.882937 8539 raft_consensus.cc:374] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.883113 8539 raft_consensus.cc:729] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.883512 8539 consensus_queue.cc:260] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.883761 8540 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:01.883952 8540 ts_tablet_manager.cc:1367] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.002s
I20240503 14:13:01.884832 8540 raft_consensus.cc:348] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.885073 8540 raft_consensus.cc:374] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.885198 8540 raft_consensus.cc:729] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.885468 8540 consensus_queue.cc:260] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.886252 8542 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:01.886432 8542 ts_tablet_manager.cc:1367] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240503 14:13:01.886893 8540 ts_tablet_manager.cc:1398] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:01.887161 8539 ts_tablet_manager.cc:1398] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:01.887198 8540 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:01.887356 8542 raft_consensus.cc:348] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:01.887545 8539 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:01.887642 8542 raft_consensus.cc:374] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.887804 8542 raft_consensus.cc:729] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.888082 8542 consensus_queue.cc:260] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:01.888671 8542 ts_tablet_manager.cc:1398] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:01.888973 8540 tablet_bootstrap.cc:654] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.892262 8539 tablet_bootstrap.cc:654] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.899014 8538 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:01.899302 8538 ts_tablet_manager.cc:1367] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.006s
I20240503 14:13:01.899819 8539 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:01.900017 8539 ts_tablet_manager.cc:1367] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20240503 14:13:01.900205 8538 raft_consensus.cc:348] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.900374 8540 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:01.900424 8538 raft_consensus.cc:374] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.900559 8538 raft_consensus.cc:729] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.900624 8540 ts_tablet_manager.cc:1367] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20240503 14:13:01.900805 8538 consensus_queue.cc:260] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.900910 8539 raft_consensus.cc:348] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.901186 8539 raft_consensus.cc:374] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.901286 8539 raft_consensus.cc:729] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.901496 8538 ts_tablet_manager.cc:1398] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:01.901554 8539 consensus_queue.cc:260] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.901959 8540 raft_consensus.cc:348] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.902196 8540 raft_consensus.cc:374] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.902335 8540 raft_consensus.cc:729] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.902501 8538 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:01.902634 8540 consensus_queue.cc:260] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.903223 8539 ts_tablet_manager.cc:1398] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:01.904079 8540 ts_tablet_manager.cc:1398] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:01.904371 8540 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:01.906953 8540 tablet_bootstrap.cc:654] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.909189 8538 tablet_bootstrap.cc:654] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.909608 8540 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:01.909745 8540 ts_tablet_manager.cc:1367] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 14:13:01.910300 8540 raft_consensus.cc:348] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:01.910475 8540 raft_consensus.cc:374] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.910573 8540 raft_consensus.cc:729] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.910790 8540 consensus_queue.cc:260] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:01.911087 8538 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:01.911255 8538 ts_tablet_manager.cc:1367] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:13:01.911899 8540 ts_tablet_manager.cc:1398] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:01.912312 8538 raft_consensus.cc:348] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:01.912515 8538 raft_consensus.cc:374] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.912602 8538 raft_consensus.cc:729] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.912863 8538 consensus_queue.cc:260] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:01.913630 8538 ts_tablet_manager.cc:1398] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:01.913996 8538 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:01.916960 8538 tablet_bootstrap.cc:654] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:01.918974 8538 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:01.919111 8538 ts_tablet_manager.cc:1367] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:01.919895 8538 raft_consensus.cc:348] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.920069 8538 raft_consensus.cc:374] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:01.920145 8538 raft_consensus.cc:729] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:01.920342 8538 consensus_queue.cc:260] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.920919 8538 ts_tablet_manager.cc:1398] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:01.953258 8470 raft_consensus.cc:482] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:01.953459 8470 raft_consensus.cc:504] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.954283 8470 leader_election.cc:288] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:01.954806 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:01.954893 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:01.955040 7189 raft_consensus.cc:2454] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:01.955183 7059 raft_consensus.cc:2454] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:01.955520 6866 leader_election.cc:302] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:01.955830 8470 raft_consensus.cc:2781] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:01.955937 8470 raft_consensus.cc:482] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:01.956027 8470 raft_consensus.cc:3037] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.958067 8470 raft_consensus.cc:504] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.958698 8470 leader_election.cc:288] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:01.959216 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:01.959282 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:01.959446 7189 raft_consensus.cc:3037] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.959568 7059 raft_consensus.cc:3037] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:01.961742 7189 raft_consensus.cc:2454] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:01.961985 7059 raft_consensus.cc:2454] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:01.962121 6866 leader_election.cc:302] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:01.962450 8470 raft_consensus.cc:2781] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:01.962586 8470 raft_consensus.cc:686] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:01.962858 8470 consensus_queue.cc:237] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:01.965844 6651 catalog_manager.cc:5484] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:02.027271 8470 raft_consensus.cc:482] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:02.027505 8470 raft_consensus.cc:504] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:02.028214 8470 leader_election.cc:288] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:02.028820 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e2d7664e6c45d69f4f8b20dcf0e4d0" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:02.028838 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e2d7664e6c45d69f4f8b20dcf0e4d0" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:02.029059 7059 raft_consensus.cc:2454] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:02.029078 7189 raft_consensus.cc:2454] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:02.029518 6867 leader_election.cc:302] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:02.029819 8470 raft_consensus.cc:2781] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:02.029920 8470 raft_consensus.cc:482] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:02.030009 8470 raft_consensus.cc:3037] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.031952 8470 raft_consensus.cc:504] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:02.032541 8470 leader_election.cc:288] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:02.033057 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e2d7664e6c45d69f4f8b20dcf0e4d0" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:02.033071 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3e2d7664e6c45d69f4f8b20dcf0e4d0" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:02.033257 7059 raft_consensus.cc:3037] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.033279 7189 raft_consensus.cc:3037] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.035552 7189 raft_consensus.cc:2454] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:02.035948 7059 raft_consensus.cc:2454] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:02.035971 6866 leader_election.cc:302] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:02.036224 8470 raft_consensus.cc:2781] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:02.036396 8470 raft_consensus.cc:686] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:02.036679 8470 consensus_queue.cc:237] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:02.039848 6651 catalog_manager.cc:5484] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:02.058720 8470 raft_consensus.cc:482] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:02.058912 8470 raft_consensus.cc:504] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:02.059773 8470 leader_election.cc:288] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:02.060420 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:02.060452 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:02.060691 7189 raft_consensus.cc:2454] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:02.060724 6799 raft_consensus.cc:2454] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:02.061178 6866 leader_election.cc:302] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:02.061455 8470 raft_consensus.cc:2781] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:02.061573 8470 raft_consensus.cc:482] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:02.061664 8470 raft_consensus.cc:3037] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.063720 8470 raft_consensus.cc:504] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:02.064291 8470 leader_election.cc:288] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:02.064862 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:02.064888 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:02.065090 7189 raft_consensus.cc:3037] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.065112 6799 raft_consensus.cc:3037] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:02.068039 7189 raft_consensus.cc:2454] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:02.068022 6799 raft_consensus.cc:2454] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:02.068567 6866 leader_election.cc:302] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:02.068923 8470 raft_consensus.cc:2781] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:02.069083 8470 raft_consensus.cc:686] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:02.069376 8470 consensus_queue.cc:237] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:02.072721 6651 catalog_manager.cc:5484] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:02.444001 8470 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:02.446447 8554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:02.446950 8554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:13:02.450567 8470 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:02.451299 8470 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:02.459448 8554 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:02.468385 8470 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:02.468588 8565 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:02.488068 8565 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
INSERT report
rows total: 2000
time total: 298.151 ms
time per row: 0.149076 ms
I20240503 14:13:02.922855 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:02.932533 7040 tablet_service.cc:1467] Processing CreateTablet for tablet cef6534b85714da5a1a066b33b903f8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.932533 6778 tablet_service.cc:1467] Processing CreateTablet for tablet cef6534b85714da5a1a066b33b903f8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.933025 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef6534b85714da5a1a066b33b903f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.933025 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef6534b85714da5a1a066b33b903f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.936720 6910 tablet_service.cc:1467] Processing CreateTablet for tablet cef6534b85714da5a1a066b33b903f8a (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.937214 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cef6534b85714da5a1a066b33b903f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.937590 7169 tablet_service.cc:1467] Processing CreateTablet for tablet a462f1d221e84693a523701a8b763b4e (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.938009 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a462f1d221e84693a523701a8b763b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.937944 7041 tablet_service.cc:1467] Processing CreateTablet for tablet a462f1d221e84693a523701a8b763b4e (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.938345 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a462f1d221e84693a523701a8b763b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.943491 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.944231 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.944545 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.944753 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.948282 6779 tablet_service.cc:1467] Processing CreateTablet for tablet a462f1d221e84693a523701a8b763b4e (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.948917 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a462f1d221e84693a523701a8b763b4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.952241 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 (DEFAULT_TABLE table=rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:02.952819 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:02.964363 8619 tablet_bootstrap.cc:492] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:02.964499 8620 tablet_bootstrap.cc:492] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:02.966866 8620 tablet_bootstrap.cc:654] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.967263 8621 tablet_bootstrap.cc:492] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:02.967902 8619 tablet_bootstrap.cc:654] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.969380 8621 tablet_bootstrap.cc:654] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.974777 8622 tablet_bootstrap.cc:492] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:02.977525 8622 tablet_bootstrap.cc:654] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.977903 8619 tablet_bootstrap.cc:492] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:02.978181 8619 ts_tablet_manager.cc:1367] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20240503 14:13:02.979347 8619 raft_consensus.cc:348] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:02.979739 8619 raft_consensus.cc:374] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.979899 8619 raft_consensus.cc:729] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.980356 8620 tablet_bootstrap.cc:492] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:02.980633 8620 ts_tablet_manager.cc:1367] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.011s
I20240503 14:13:02.980640 8619 consensus_queue.cc:260] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:02.981295 8619 ts_tablet_manager.cc:1398] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:02.981642 8619 tablet_bootstrap.cc:492] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:02.981752 8620 raft_consensus.cc:348] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:02.982065 8620 raft_consensus.cc:374] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.982185 8620 raft_consensus.cc:729] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.982524 8620 consensus_queue.cc:260] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:02.983302 8620 ts_tablet_manager.cc:1398] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:02.983765 8619 tablet_bootstrap.cc:654] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.988509 8622 tablet_bootstrap.cc:492] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:02.988765 8622 ts_tablet_manager.cc:1367] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20240503 14:13:02.989912 8621 tablet_bootstrap.cc:492] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:02.990123 8621 ts_tablet_manager.cc:1367] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20240503 14:13:02.991410 8621 raft_consensus.cc:348] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:02.991808 8621 raft_consensus.cc:374] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.991677 8622 raft_consensus.cc:348] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:02.991899 8622 raft_consensus.cc:374] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.991945 8621 raft_consensus.cc:729] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.992030 8619 tablet_bootstrap.cc:492] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:02.992028 8622 raft_consensus.cc:729] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.992226 8619 ts_tablet_manager.cc:1367] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240503 14:13:02.992238 8621 consensus_queue.cc:260] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:02.992300 8622 consensus_queue.cc:260] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:02.992992 8622 ts_tablet_manager.cc:1398] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:02.992990 8619 raft_consensus.cc:348] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:02.993245 8619 raft_consensus.cc:374] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:02.993311 8622 tablet_bootstrap.cc:492] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:02.993346 8619 raft_consensus.cc:729] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:02.993767 8621 ts_tablet_manager.cc:1398] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:02.993680 8619 consensus_queue.cc:260] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:02.994529 8619 ts_tablet_manager.cc:1398] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:02.994894 8619 tablet_bootstrap.cc:492] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:02.994911 8621 tablet_bootstrap.cc:492] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:02.995574 8622 tablet_bootstrap.cc:654] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:02.996615 8619 tablet_bootstrap.cc:654] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:03.000908 8621 tablet_bootstrap.cc:654] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:03.005273 8619 tablet_bootstrap.cc:492] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:03.005604 8619 ts_tablet_manager.cc:1367] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240503 14:13:03.006559 8619 raft_consensus.cc:348] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:03.006860 8619 raft_consensus.cc:374] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:03.006997 8619 raft_consensus.cc:729] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:03.007299 8619 consensus_queue.cc:260] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:03.008090 8619 ts_tablet_manager.cc:1398] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:03.010713 8621 tablet_bootstrap.cc:492] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:03.010973 8621 ts_tablet_manager.cc:1367] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240503 14:13:03.011451 8622 tablet_bootstrap.cc:492] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:03.012006 8621 raft_consensus.cc:348] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:03.012198 8621 raft_consensus.cc:374] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:03.012306 8621 raft_consensus.cc:729] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:03.012548 8621 consensus_queue.cc:260] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:03.012868 8622 ts_tablet_manager.cc:1367] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.009s
I20240503 14:13:03.013088 8621 ts_tablet_manager.cc:1398] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:03.013761 8622 raft_consensus.cc:348] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:03.013976 8622 raft_consensus.cc:374] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:03.014060 8622 raft_consensus.cc:729] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:03.014302 8622 consensus_queue.cc:260] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:03.014775 8622 ts_tablet_manager.cc:1398] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:03.015136 8622 tablet_bootstrap.cc:492] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:03.016633 8622 tablet_bootstrap.cc:654] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:03.018548 8622 tablet_bootstrap.cc:492] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:03.018733 8622 ts_tablet_manager.cc:1367] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:03.019376 8622 raft_consensus.cc:348] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:03.019547 8622 raft_consensus.cc:374] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:03.019694 8622 raft_consensus.cc:729] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:03.019951 8622 consensus_queue.cc:260] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:03.020439 8622 ts_tablet_manager.cc:1398] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:03.031915 8248 raft_consensus.cc:482] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:03.032114 8248 raft_consensus.cc:504] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:03.032678 8248 leader_election.cc:288] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:03.033208 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef6534b85714da5a1a066b33b903f8a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:03.033237 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef6534b85714da5a1a066b33b903f8a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:03.033464 6800 raft_consensus.cc:2454] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:03.033516 7059 raft_consensus.cc:2454] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:03.033946 6865 leader_election.cc:302] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:03.034281 8248 raft_consensus.cc:2781] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:03.034389 8248 raft_consensus.cc:482] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:03.034518 8248 raft_consensus.cc:3037] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.036643 8248 raft_consensus.cc:504] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:03.037220 8248 leader_election.cc:288] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:03.037618 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef6534b85714da5a1a066b33b903f8a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:03.037668 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cef6534b85714da5a1a066b33b903f8a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:03.037829 6800 raft_consensus.cc:3037] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.037853 7059 raft_consensus.cc:3037] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.039817 7059 raft_consensus.cc:2454] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:03.039855 6800 raft_consensus.cc:2454] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:03.040259 6867 leader_election.cc:302] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:03.040619 8248 raft_consensus.cc:2781] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:03.040794 8248 raft_consensus.cc:686] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:03.041086 8248 consensus_queue.cc:237] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:03.044795 6659 catalog_manager.cc:5484] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:03.076174 8335 raft_consensus.cc:482] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:03.076391 8335 raft_consensus.cc:504] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:03.077085 8335 leader_election.cc:288] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:03.077625 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f8e607d4fa40efb8d97d1c11ccf2c1" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:03.077682 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f8e607d4fa40efb8d97d1c11ccf2c1" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:03.077881 6929 raft_consensus.cc:2454] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:03.077965 7059 raft_consensus.cc:2454] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:03.078363 6734 leader_election.cc:302] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:03.078666 8335 raft_consensus.cc:2781] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:03.078801 8335 raft_consensus.cc:482] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:03.078883 8335 raft_consensus.cc:3037] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.080807 8335 raft_consensus.cc:504] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:03.081414 8335 leader_election.cc:288] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:03.081908 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f8e607d4fa40efb8d97d1c11ccf2c1" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:03.082005 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f8e607d4fa40efb8d97d1c11ccf2c1" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:03.082146 6929 raft_consensus.cc:3037] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.082213 7059 raft_consensus.cc:3037] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.084121 6929 raft_consensus.cc:2454] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:03.084376 7059 raft_consensus.cc:2454] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:03.084623 6734 leader_election.cc:302] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:03.084995 8335 raft_consensus.cc:2781] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:03.085402 8335 raft_consensus.cc:686] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:03.085721 8335 consensus_queue.cc:237] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:03.088924 6659 catalog_manager.cc:5484] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:03.094620 8312 raft_consensus.cc:482] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:03.094811 8312 raft_consensus.cc:504] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:03.095441 8312 leader_election.cc:288] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:03.096019 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a462f1d221e84693a523701a8b763b4e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:03.096086 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a462f1d221e84693a523701a8b763b4e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:03.096254 6800 raft_consensus.cc:2454] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:03.096314 7059 raft_consensus.cc:2454] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:03.096673 7125 leader_election.cc:302] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:03.096940 8312 raft_consensus.cc:2781] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:03.097052 8312 raft_consensus.cc:482] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:03.097122 8312 raft_consensus.cc:3037] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.098989 8312 raft_consensus.cc:504] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:03.099647 8312 leader_election.cc:288] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:03.100179 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a462f1d221e84693a523701a8b763b4e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:03.100210 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a462f1d221e84693a523701a8b763b4e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:03.100378 6800 raft_consensus.cc:3037] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.100399 7059 raft_consensus.cc:3037] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:03.102267 7059 raft_consensus.cc:2454] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:03.102397 6800 raft_consensus.cc:2454] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:03.102774 7127 leader_election.cc:302] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:03.103099 8312 raft_consensus.cc:2781] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:03.103514 8312 raft_consensus.cc:686] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:03.103919 8312 consensus_queue.cc:237] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:03.107317 6659 catalog_manager.cc:5484] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:03.431701 8248 consensus_queue.cc:1035] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:03.438803 8248 consensus_queue.cc:1035] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:03.522066 8637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:03.522436 8637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:03.530226 8637 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240503 14:13:03.608003 8635 consensus_queue.cc:1035] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:03.623100 8635 consensus_queue.cc:1035] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:03.626669 8633 consensus_queue.cc:1035] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:03.634229 8335 consensus_queue.cc:1035] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 320.173 ms
time per row: 0.160086 ms
I20240503 14:13:04.012004 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:04.022375 7169 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5781b2310f4814aaebdcb715b7ddac (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.022897 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5781b2310f4814aaebdcb715b7ddac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.023195 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 847ef240dcc64fb794ffcc06e483dc08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.023572 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847ef240dcc64fb794ffcc06e483dc08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.022521 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5781b2310f4814aaebdcb715b7ddac (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.023689 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 847ef240dcc64fb794ffcc06e483dc08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.023947 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5781b2310f4814aaebdcb715b7ddac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.024168 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847ef240dcc64fb794ffcc06e483dc08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.025979 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 34f1ed0cfa8948048b3762919481367c (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.026383 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f1ed0cfa8948048b3762919481367c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.028295 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5781b2310f4814aaebdcb715b7ddac (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.028425 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 847ef240dcc64fb794ffcc06e483dc08 (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.028733 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5781b2310f4814aaebdcb715b7ddac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.028836 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847ef240dcc64fb794ffcc06e483dc08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.032054 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 34f1ed0cfa8948048b3762919481367c (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.032142 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 34f1ed0cfa8948048b3762919481367c (DEFAULT_TABLE table=rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:04.032474 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f1ed0cfa8948048b3762919481367c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.032662 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f1ed0cfa8948048b3762919481367c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:04.044854 8706 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:04.046841 8706 tablet_bootstrap.cc:654] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.050967 8709 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:04.053283 8706 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:04.053398 8708 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:04.053673 8707 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:04.054118 8706 ts_tablet_manager.cc:1367] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:04.055205 8707 tablet_bootstrap.cc:654] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.055095 8706 raft_consensus.cc:348] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.055318 8708 tablet_bootstrap.cc:654] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.055333 8706 raft_consensus.cc:374] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.055424 8706 raft_consensus.cc:729] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.055749 8706 consensus_queue.cc:260] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.056689 8709 tablet_bootstrap.cc:654] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.057785 8706 ts_tablet_manager.cc:1398] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:04.058301 8706 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:04.061080 8707 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:04.061241 8707 ts_tablet_manager.cc:1367] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 14:13:04.061460 8706 tablet_bootstrap.cc:654] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.062139 8707 raft_consensus.cc:348] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.062367 8707 raft_consensus.cc:374] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.062471 8707 raft_consensus.cc:729] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.062700 8707 consensus_queue.cc:260] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.063267 8707 ts_tablet_manager.cc:1398] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:04.063565 8707 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:04.065326 8707 tablet_bootstrap.cc:654] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.068459 8709 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:04.068639 8709 ts_tablet_manager.cc:1367] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240503 14:13:04.069517 8709 raft_consensus.cc:348] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.069617 8708 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:04.069701 8709 raft_consensus.cc:374] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.069778 8708 ts_tablet_manager.cc:1367] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.008s
I20240503 14:13:04.069792 8709 raft_consensus.cc:729] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.070026 8709 consensus_queue.cc:260] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.070638 8709 ts_tablet_manager.cc:1398] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:04.070989 8709 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:04.071560 8706 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:04.071761 8706 ts_tablet_manager.cc:1367] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.003s
I20240503 14:13:04.072638 8709 tablet_bootstrap.cc:654] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.072568 8706 raft_consensus.cc:348] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.072916 8706 raft_consensus.cc:374] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.073066 8706 raft_consensus.cc:729] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.071043 8708 raft_consensus.cc:348] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.073391 8708 raft_consensus.cc:374] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.073514 8708 raft_consensus.cc:729] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.073833 8708 consensus_queue.cc:260] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.074429 8706 consensus_queue.cc:260] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.076076 8706 ts_tablet_manager.cc:1398] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:04.077509 8708 ts_tablet_manager.cc:1398] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 14:13:04.077904 8708 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:04.079782 8708 tablet_bootstrap.cc:654] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.083863 8709 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:04.084092 8709 ts_tablet_manager.cc:1367] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20240503 14:13:04.084906 8709 raft_consensus.cc:348] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.085184 8709 raft_consensus.cc:374] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.085270 8709 raft_consensus.cc:729] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.085530 8709 consensus_queue.cc:260] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.086062 8709 ts_tablet_manager.cc:1398] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:04.086356 8709 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:04.088016 8709 tablet_bootstrap.cc:654] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:04.089020 8707 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:04.089067 8708 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:04.089275 8708 ts_tablet_manager.cc:1367] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240503 14:13:04.089409 8707 ts_tablet_manager.cc:1367] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.010s
I20240503 14:13:04.090101 8708 raft_consensus.cc:348] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.090346 8708 raft_consensus.cc:374] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.090435 8708 raft_consensus.cc:729] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.090744 8708 consensus_queue.cc:260] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.090682 8707 raft_consensus.cc:348] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.091173 8707 raft_consensus.cc:374] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.091317 8707 raft_consensus.cc:729] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.091755 8707 consensus_queue.cc:260] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.092919 8707 ts_tablet_manager.cc:1398] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:04.093262 8708 ts_tablet_manager.cc:1398] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:04.097249 8709 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:04.097415 8709 ts_tablet_manager.cc:1367] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 14:13:04.098167 8709 raft_consensus.cc:348] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.098402 8709 raft_consensus.cc:374] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:04.098497 8709 raft_consensus.cc:729] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:04.098784 8709 consensus_queue.cc:260] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.099361 8709 ts_tablet_manager.cc:1398] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:04.257586 8335 raft_consensus.cc:482] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:04.257812 8335 raft_consensus.cc:504] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.258508 8335 leader_election.cc:288] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:04.259073 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5781b2310f4814aaebdcb715b7ddac" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:04.259145 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5781b2310f4814aaebdcb715b7ddac" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:04.259347 6931 raft_consensus.cc:2454] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:04.259362 7189 raft_consensus.cc:2454] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:04.259881 6735 leader_election.cc:302] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:04.260133 8335 raft_consensus.cc:2781] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:04.260255 8335 raft_consensus.cc:482] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:04.260347 8335 raft_consensus.cc:3037] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.262365 8335 raft_consensus.cc:504] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.262954 8335 leader_election.cc:288] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:04.263540 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5781b2310f4814aaebdcb715b7ddac" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:04.263556 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5781b2310f4814aaebdcb715b7ddac" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:04.263769 6931 raft_consensus.cc:3037] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.263810 7189 raft_consensus.cc:3037] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.265676 7189 raft_consensus.cc:2454] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:04.265687 6931 raft_consensus.cc:2454] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:04.266078 6734 leader_election.cc:302] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:04.266361 8335 raft_consensus.cc:2781] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:04.266489 8335 raft_consensus.cc:686] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:04.266752 8335 consensus_queue.cc:237] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:04.270332 6650 catalog_manager.cc:5484] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:04.290810 8605 raft_consensus.cc:482] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:04.291028 8605 raft_consensus.cc:504] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.291896 8605 leader_election.cc:288] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:04.292248 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f1ed0cfa8948048b3762919481367c" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:04.292533 7060 raft_consensus.cc:2454] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:04.292479 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f1ed0cfa8948048b3762919481367c" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:04.292783 7189 raft_consensus.cc:2454] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:04.293010 6867 leader_election.cc:302] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:04.293283 8248 raft_consensus.cc:2781] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:04.293375 8248 raft_consensus.cc:482] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:04.293478 8248 raft_consensus.cc:3037] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.295363 8248 raft_consensus.cc:504] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.296043 8248 leader_election.cc:288] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:04.296590 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f1ed0cfa8948048b3762919481367c" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:04.296684 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34f1ed0cfa8948048b3762919481367c" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:04.296808 7060 raft_consensus.cc:3037] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.296908 7189 raft_consensus.cc:3037] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.299047 7060 raft_consensus.cc:2454] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:04.299556 8312 raft_consensus.cc:482] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:04.299803 8312 raft_consensus.cc:504] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.299937 7189 raft_consensus.cc:2454] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:04.300453 6866 leader_election.cc:302] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:04.300598 8312 leader_election.cc:288] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:04.300777 8248 raft_consensus.cc:2781] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:04.300971 8248 raft_consensus.cc:686] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:04.301244 8248 consensus_queue.cc:237] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:04.301277 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847ef240dcc64fb794ffcc06e483dc08" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:04.301524 6799 raft_consensus.cc:2454] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:04.301489 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847ef240dcc64fb794ffcc06e483dc08" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:04.301694 7060 raft_consensus.cc:2454] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:04.301955 7125 leader_election.cc:302] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:04.302232 8312 raft_consensus.cc:2781] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:04.302328 8312 raft_consensus.cc:482] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:04.302436 8312 raft_consensus.cc:3037] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.305163 8312 raft_consensus.cc:504] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.305621 6650 catalog_manager.cc:5484] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:04.306361 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847ef240dcc64fb794ffcc06e483dc08" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:04.306567 7060 raft_consensus.cc:3037] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.309121 7060 raft_consensus.cc:2454] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:04.310109 7127 leader_election.cc:302] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:04.310954 8312 leader_election.cc:288] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:04.311259 8312 raft_consensus.cc:2781] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:04.311498 8312 raft_consensus.cc:686] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:04.314179 8312 consensus_queue.cc:237] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:04.318204 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "847ef240dcc64fb794ffcc06e483dc08" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:04.318508 6799 raft_consensus.cc:3037] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:04.322525 6799 raft_consensus.cc:2454] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:04.327739 6650 catalog_manager.cc:5484] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:04.702672 8725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:04.703094 8725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 14:13:04.703414 8248 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:04.713783 8605 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:04.715525 8725 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:04.716413 8721 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:04.742730 8312 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_11'
I20240503 14:13:04.816504 8335 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:04.836223 8687 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:04.984899 6865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.1:34101, user_credentials={real_user=slave}} blocked reactor thread for 64591.7us
INSERT report
rows total: 2000
time total: 292.214 ms
time per row: 0.146107 ms
I20240503 14:13:05.198729 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:05.208412 6778 tablet_service.cc:1467] Processing CreateTablet for tablet cc98f88f68af446696b49b1ddd9bde51 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.208737 6910 tablet_service.cc:1467] Processing CreateTablet for tablet cc98f88f68af446696b49b1ddd9bde51 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.208976 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc98f88f68af446696b49b1ddd9bde51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.208866 7171 tablet_service.cc:1467] Processing CreateTablet for tablet cc98f88f68af446696b49b1ddd9bde51 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.209173 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc98f88f68af446696b49b1ddd9bde51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.209290 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc98f88f68af446696b49b1ddd9bde51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.212934 6911 tablet_service.cc:1467] Processing CreateTablet for tablet b045a23f3e584ee1a144645d93cf6982 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.213363 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b045a23f3e584ee1a144645d93cf6982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.213335 6779 tablet_service.cc:1467] Processing CreateTablet for tablet b045a23f3e584ee1a144645d93cf6982 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.213438 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 9d58cbd2efb746148f7deb34a38cd442 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.213721 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b045a23f3e584ee1a144645d93cf6982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.213873 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d58cbd2efb746148f7deb34a38cd442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.216698 7040 tablet_service.cc:1467] Processing CreateTablet for tablet b045a23f3e584ee1a144645d93cf6982 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.217130 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b045a23f3e584ee1a144645d93cf6982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.220032 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 9d58cbd2efb746148f7deb34a38cd442 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.220467 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d58cbd2efb746148f7deb34a38cd442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.224357 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 9d58cbd2efb746148f7deb34a38cd442 (DEFAULT_TABLE table=rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:05.224931 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d58cbd2efb746148f7deb34a38cd442. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:05.225512 8794 tablet_bootstrap.cc:492] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:05.225924 8795 tablet_bootstrap.cc:492] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:05.227981 8795 tablet_bootstrap.cc:654] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.227981 8794 tablet_bootstrap.cc:654] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.237375 8798 tablet_bootstrap.cc:492] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:05.237974 8794 tablet_bootstrap.cc:492] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:05.238209 8794 ts_tablet_manager.cc:1367] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 14:13:05.239080 8794 raft_consensus.cc:348] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.239323 8794 raft_consensus.cc:374] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.239495 8794 raft_consensus.cc:729] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.239792 8794 consensus_queue.cc:260] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.240049 8795 tablet_bootstrap.cc:492] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:05.240245 8795 ts_tablet_manager.cc:1367] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240503 14:13:05.240399 8794 ts_tablet_manager.cc:1398] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:05.241238 8795 raft_consensus.cc:348] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.241500 8794 tablet_bootstrap.cc:492] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:05.241508 8795 raft_consensus.cc:374] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.241600 8795 raft_consensus.cc:729] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.241817 8795 consensus_queue.cc:260] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.242524 8795 ts_tablet_manager.cc:1398] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:05.242823 8795 tablet_bootstrap.cc:492] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:05.244494 8795 tablet_bootstrap.cc:654] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.252851 8799 tablet_bootstrap.cc:492] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:05.253798 8794 tablet_bootstrap.cc:654] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.255227 8798 tablet_bootstrap.cc:654] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.256100 8799 tablet_bootstrap.cc:654] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.256968 8795 tablet_bootstrap.cc:492] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:05.257211 8795 ts_tablet_manager.cc:1367] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.010s
I20240503 14:13:05.258119 8795 raft_consensus.cc:348] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.258435 8795 raft_consensus.cc:374] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.258567 8795 raft_consensus.cc:729] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.258937 8795 consensus_queue.cc:260] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.259555 8795 ts_tablet_manager.cc:1398] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:05.259212 8794 tablet_bootstrap.cc:492] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:05.261369 8794 ts_tablet_manager.cc:1367] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.002s
I20240503 14:13:05.263164 8794 raft_consensus.cc:348] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.263571 8794 raft_consensus.cc:374] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.263751 8794 raft_consensus.cc:729] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.266112 8794 consensus_queue.cc:260] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.267174 8794 ts_tablet_manager.cc:1398] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:05.290091 8799 tablet_bootstrap.cc:492] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:05.290256 8798 tablet_bootstrap.cc:492] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:05.290330 8799 ts_tablet_manager.cc:1367] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.000s
I20240503 14:13:05.290459 8798 ts_tablet_manager.cc:1367] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.054s user 0.018s sys 0.014s
I20240503 14:13:05.291224 8798 raft_consensus.cc:348] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.291219 8799 raft_consensus.cc:348] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.291440 8799 raft_consensus.cc:374] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.291440 8798 raft_consensus.cc:374] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.291540 8798 raft_consensus.cc:729] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.291527 8799 raft_consensus.cc:729] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.291852 8799 consensus_queue.cc:260] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.291852 8798 consensus_queue.cc:260] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.292449 8799 ts_tablet_manager.cc:1398] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:05.292596 8798 ts_tablet_manager.cc:1398] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:05.292953 8799 tablet_bootstrap.cc:492] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:05.292965 8798 tablet_bootstrap.cc:492] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:05.294858 8799 tablet_bootstrap.cc:654] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.296679 8798 tablet_bootstrap.cc:654] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.304975 8798 tablet_bootstrap.cc:492] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:05.305213 8798 ts_tablet_manager.cc:1367] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20240503 14:13:05.305311 8799 tablet_bootstrap.cc:492] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:05.305487 8799 ts_tablet_manager.cc:1367] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20240503 14:13:05.306257 8799 raft_consensus.cc:348] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.306205 8798 raft_consensus.cc:348] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.306473 8799 raft_consensus.cc:374] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.306481 8798 raft_consensus.cc:374] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.306574 8798 raft_consensus.cc:729] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.306569 8799 raft_consensus.cc:729] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.306841 8799 consensus_queue.cc:260] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.306883 8798 consensus_queue.cc:260] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.307427 8799 ts_tablet_manager.cc:1398] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:05.307529 8798 ts_tablet_manager.cc:1398] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:13:05.307911 8798 tablet_bootstrap.cc:492] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:05.309454 8798 tablet_bootstrap.cc:654] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:05.312007 8798 tablet_bootstrap.cc:492] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:05.312186 8798 ts_tablet_manager.cc:1367] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240503 14:13:05.313067 8798 raft_consensus.cc:348] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.313292 8798 raft_consensus.cc:374] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:05.313395 8798 raft_consensus.cc:729] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:05.313690 8798 consensus_queue.cc:260] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.314319 8798 ts_tablet_manager.cc:1398] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:05.355482 8765 raft_consensus.cc:482] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:05.355749 8765 raft_consensus.cc:504] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.356644 8765 leader_election.cc:288] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:05.357208 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b045a23f3e584ee1a144645d93cf6982" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:05.357221 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b045a23f3e584ee1a144645d93cf6982" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:05.357437 6799 raft_consensus.cc:2454] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:05.357482 7060 raft_consensus.cc:2454] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:05.357941 6865 leader_election.cc:302] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:05.358281 8765 raft_consensus.cc:2781] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:05.358389 8765 raft_consensus.cc:482] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:05.358480 8765 raft_consensus.cc:3037] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.360417 8765 raft_consensus.cc:504] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.361037 8765 leader_election.cc:288] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:05.361543 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b045a23f3e584ee1a144645d93cf6982" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:05.361591 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b045a23f3e584ee1a144645d93cf6982" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:05.361758 7060 raft_consensus.cc:3037] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.361801 6799 raft_consensus.cc:3037] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.364146 6799 raft_consensus.cc:2454] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:05.364177 7060 raft_consensus.cc:2454] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:05.364580 6865 leader_election.cc:302] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:05.364872 8765 raft_consensus.cc:2781] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:05.365023 8765 raft_consensus.cc:686] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:05.365305 8765 consensus_queue.cc:237] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.368693 6658 catalog_manager.cc:5484] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:05.418269 8765 raft_consensus.cc:482] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:05.418501 8765 raft_consensus.cc:504] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.419353 8765 leader_election.cc:288] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:05.419867 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc98f88f68af446696b49b1ddd9bde51" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:05.419943 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc98f88f68af446696b49b1ddd9bde51" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:05.420094 6799 raft_consensus.cc:2454] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:05.420174 7189 raft_consensus.cc:2454] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:05.420682 6866 leader_election.cc:302] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:05.420970 8765 raft_consensus.cc:2781] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:05.421051 8765 raft_consensus.cc:482] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:05.421145 8765 raft_consensus.cc:3037] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.423199 8765 raft_consensus.cc:504] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.424420 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc98f88f68af446696b49b1ddd9bde51" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:05.424533 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc98f88f68af446696b49b1ddd9bde51" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:05.424674 7189 raft_consensus.cc:3037] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.424775 6799 raft_consensus.cc:3037] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.426972 7189 raft_consensus.cc:2454] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:05.427083 6799 raft_consensus.cc:2454] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:05.427421 6866 leader_election.cc:302] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:05.427809 8772 raft_consensus.cc:2781] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:05.427949 8772 raft_consensus.cc:686] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:05.423938 8765 leader_election.cc:288] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:05.428258 8772 consensus_queue.cc:237] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:05.432291 6658 catalog_manager.cc:5484] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:05.612967 8701 raft_consensus.cc:482] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:05.613183 8701 raft_consensus.cc:504] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.613955 8701 leader_election.cc:288] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:05.614408 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d58cbd2efb746148f7deb34a38cd442" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:05.614439 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d58cbd2efb746148f7deb34a38cd442" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:05.614621 7189 raft_consensus.cc:2454] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:05.614691 6799 raft_consensus.cc:2454] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:05.615088 6996 leader_election.cc:302] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:05.615365 8701 raft_consensus.cc:2781] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:05.615484 8701 raft_consensus.cc:482] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:05.615571 8701 raft_consensus.cc:3037] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.617493 8701 raft_consensus.cc:504] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.618144 8701 leader_election.cc:288] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:05.618582 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d58cbd2efb746148f7deb34a38cd442" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:05.618796 7189 raft_consensus.cc:3037] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.618731 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d58cbd2efb746148f7deb34a38cd442" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:05.618971 6799 raft_consensus.cc:3037] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:05.620854 7189 raft_consensus.cc:2454] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:05.620990 6799 raft_consensus.cc:2454] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:05.621241 6996 leader_election.cc:302] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:05.621531 8701 raft_consensus.cc:2781] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:05.621932 8701 raft_consensus.cc:686] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:05.622191 8701 consensus_queue.cc:237] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:05.625124 6658 catalog_manager.cc:5484] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:05.859972 8248 consensus_queue.cc:1035] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:05.868978 8248 consensus_queue.cc:1035] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:05.917433 8248 consensus_queue.cc:1035] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:05.922070 8765 consensus_queue.cc:1035] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:06.041913 8701 consensus_queue.cc:1035] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:06.047688 8809 consensus_queue.cc:1035] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:06.090235 8811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:06.090580 8811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:06.099718 8811 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
W20240503 14:13:06.303820 6867 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 44676.6us
INSERT report
rows total: 2000
time total: 261.646 ms
time per row: 0.130823 ms
I20240503 14:13:06.516464 6657 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:06.526211 7171 tablet_service.cc:1467] Processing CreateTablet for tablet f5738d4265f941519b90b07334d06af9 (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.526245 7040 tablet_service.cc:1467] Processing CreateTablet for tablet f5738d4265f941519b90b07334d06af9 (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.526728 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5738d4265f941519b90b07334d06af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.526721 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5738d4265f941519b90b07334d06af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.527567 6910 tablet_service.cc:1467] Processing CreateTablet for tablet f5738d4265f941519b90b07334d06af9 (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.528074 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5738d4265f941519b90b07334d06af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.528208 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 9027b8bfee0443aab7420df3f7257bcb (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.528559 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9027b8bfee0443aab7420df3f7257bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.529106 6909 tablet_service.cc:1467] Processing CreateTablet for tablet a98e955a2faf4fc889ec01bd148dd354 (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.529489 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98e955a2faf4fc889ec01bd148dd354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.532584 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 9027b8bfee0443aab7420df3f7257bcb (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.532974 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9027b8bfee0443aab7420df3f7257bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.533464 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 9027b8bfee0443aab7420df3f7257bcb (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.533885 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9027b8bfee0443aab7420df3f7257bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.536361 6779 tablet_service.cc:1467] Processing CreateTablet for tablet a98e955a2faf4fc889ec01bd148dd354 (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.536881 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98e955a2faf4fc889ec01bd148dd354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.539405 7169 tablet_service.cc:1467] Processing CreateTablet for tablet a98e955a2faf4fc889ec01bd148dd354 (DEFAULT_TABLE table=rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:06.540099 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98e955a2faf4fc889ec01bd148dd354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:06.544494 8879 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:06.547204 8880 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:06.548137 8879 tablet_bootstrap.cc:654] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.549683 8880 tablet_bootstrap.cc:654] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.559548 8880 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:06.559892 8880 ts_tablet_manager.cc:1367] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240503 14:13:06.561092 8880 raft_consensus.cc:348] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.562212 8880 raft_consensus.cc:374] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.562402 8880 raft_consensus.cc:729] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.562942 8881 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:06.562801 8880 consensus_queue.cc:260] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.563911 8880 ts_tablet_manager.cc:1398] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:06.564275 8880 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:06.567389 8881 tablet_bootstrap.cc:654] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.568262 8884 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:06.570360 8880 tablet_bootstrap.cc:654] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.571311 8881 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:06.571491 8879 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:06.571511 8881 ts_tablet_manager.cc:1367] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:06.571669 8879 ts_tablet_manager.cc:1367] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.010s
I20240503 14:13:06.572155 8880 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:06.572314 8880 ts_tablet_manager.cc:1367] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 14:13:06.572341 8881 raft_consensus.cc:348] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.572574 8881 raft_consensus.cc:374] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.572459 8879 raft_consensus.cc:348] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.572659 8879 raft_consensus.cc:374] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.572712 8881 raft_consensus.cc:729] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.572762 8879 raft_consensus.cc:729] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.573043 8880 raft_consensus.cc:348] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.573045 8881 consensus_queue.cc:260] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.573330 8880 raft_consensus.cc:374] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.573233 8879 consensus_queue.cc:260] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.573621 8880 raft_consensus.cc:729] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.573875 8879 ts_tablet_manager.cc:1398] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:06.573953 8880 consensus_queue.cc:260] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.574942 8879 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:06.576478 8880 ts_tablet_manager.cc:1398] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:13:06.576866 8880 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:06.578210 8879 tablet_bootstrap.cc:654] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.578526 8880 tablet_bootstrap.cc:654] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.579578 8884 tablet_bootstrap.cc:654] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.583536 8879 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:06.583753 8879 ts_tablet_manager.cc:1367] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240503 14:13:06.584597 8879 raft_consensus.cc:348] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.584945 8879 raft_consensus.cc:374] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.585078 8879 raft_consensus.cc:729] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.585423 8879 consensus_queue.cc:260] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.586167 8879 ts_tablet_manager.cc:1398] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:06.586221 8881 ts_tablet_manager.cc:1398] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 14:13:06.586560 8879 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:06.587240 8880 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:06.587466 8880 ts_tablet_manager.cc:1367] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240503 14:13:06.588483 8879 tablet_bootstrap.cc:654] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.588436 8880 raft_consensus.cc:348] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.588666 8880 raft_consensus.cc:374] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.588774 8880 raft_consensus.cc:729] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.589071 8880 consensus_queue.cc:260] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.589774 8880 ts_tablet_manager.cc:1398] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:06.591468 8884 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:06.591719 8884 ts_tablet_manager.cc:1367] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20240503 14:13:06.592590 8884 raft_consensus.cc:348] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.592820 8884 raft_consensus.cc:374] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.592936 8884 raft_consensus.cc:729] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.593173 8884 consensus_queue.cc:260] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.593416 8881 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:06.593735 8884 ts_tablet_manager.cc:1398] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:06.597672 8881 tablet_bootstrap.cc:654] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:06.600078 8881 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:06.600317 8881 ts_tablet_manager.cc:1367] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240503 14:13:06.600400 8879 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:06.600597 8879 ts_tablet_manager.cc:1367] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240503 14:13:06.601178 8881 raft_consensus.cc:348] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.601438 8881 raft_consensus.cc:374] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.601527 8881 raft_consensus.cc:729] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.601438 8879 raft_consensus.cc:348] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.601681 8879 raft_consensus.cc:374] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:06.601804 8879 raft_consensus.cc:729] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:06.601845 8881 consensus_queue.cc:260] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.602077 8879 consensus_queue.cc:260] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.602645 8879 ts_tablet_manager.cc:1398] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:06.603283 8881 ts_tablet_manager.cc:1398] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:06.703740 8248 raft_consensus.cc:482] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:06.703958 8248 raft_consensus.cc:504] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.704716 8248 leader_election.cc:288] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:06.705334 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:06.705348 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:06.705557 7059 raft_consensus.cc:2454] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:06.705593 7189 raft_consensus.cc:2454] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:06.706132 6866 leader_election.cc:302] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:06.706467 8248 raft_consensus.cc:2781] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:06.706607 8248 raft_consensus.cc:482] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:06.706723 8248 raft_consensus.cc:3037] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.709215 8248 raft_consensus.cc:504] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.709823 8248 leader_election.cc:288] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:06.710346 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:06.710347 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:06.710537 7059 raft_consensus.cc:3037] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.710588 7189 raft_consensus.cc:3037] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.712908 7189 raft_consensus.cc:2454] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:06.713447 7059 raft_consensus.cc:2454] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:06.713526 6866 leader_election.cc:302] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:06.713912 8863 raft_consensus.cc:2781] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:06.714080 8863 raft_consensus.cc:686] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:06.714442 8863 consensus_queue.cc:237] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:06.718569 8863 raft_consensus.cc:482] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:06.718551 6651 catalog_manager.cc:5484] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:06.718752 8863 raft_consensus.cc:504] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.719352 8863 leader_election.cc:288] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:06.720026 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5738d4265f941519b90b07334d06af9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:06.720333 7059 raft_consensus.cc:2454] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:06.721135 6867 leader_election.cc:302] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:06.721988 8863 raft_consensus.cc:2781] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:06.722100 8863 raft_consensus.cc:482] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:06.722204 8863 raft_consensus.cc:3037] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.724354 8863 raft_consensus.cc:504] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.725069 8863 leader_election.cc:288] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:06.725536 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5738d4265f941519b90b07334d06af9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:06.725548 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5738d4265f941519b90b07334d06af9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:06.725770 7059 raft_consensus.cc:3037] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.725785 7190 raft_consensus.cc:3037] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.725559 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5738d4265f941519b90b07334d06af9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:06.728107 7059 raft_consensus.cc:2454] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:06.728714 7190 raft_consensus.cc:2454] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:06.729264 6867 leader_election.cc:302] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:06.729528 8863 raft_consensus.cc:2781] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:06.729696 8863 raft_consensus.cc:686] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:06.729995 8863 consensus_queue.cc:237] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.733461 6651 catalog_manager.cc:5484] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:06.792088 8823 raft_consensus.cc:482] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:06.792279 8823 raft_consensus.cc:504] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.793126 8823 leader_election.cc:288] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:06.793599 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98e955a2faf4fc889ec01bd148dd354" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:06.793665 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98e955a2faf4fc889ec01bd148dd354" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:06.793821 7190 raft_consensus.cc:2454] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:06.793910 6929 raft_consensus.cc:2454] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:06.794224 6735 leader_election.cc:302] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:06.794584 8823 raft_consensus.cc:2781] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:06.794692 8823 raft_consensus.cc:482] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:06.794777 8823 raft_consensus.cc:3037] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.796713 8823 raft_consensus.cc:504] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.797286 8823 leader_election.cc:288] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:06.797906 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98e955a2faf4fc889ec01bd148dd354" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:06.797919 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98e955a2faf4fc889ec01bd148dd354" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:06.798130 7190 raft_consensus.cc:3037] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.798130 6929 raft_consensus.cc:3037] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:06.801091 7190 raft_consensus.cc:2454] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:06.801087 6929 raft_consensus.cc:2454] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:06.801640 6735 leader_election.cc:302] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:06.801975 8823 raft_consensus.cc:2781] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:06.802147 8823 raft_consensus.cc:686] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:06.802467 8823 consensus_queue.cc:237] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:06.806025 6651 catalog_manager.cc:5484] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:07.188002 8863 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.195268 8248 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.196614 8248 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.203627 8248 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:07.225082 8895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:07.225467 8895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:07.237726 8895 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 14:13:07.266875 8823 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:07.275756 8823 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 240.564 ms
time per row: 0.120282 ms
I20240503 14:13:07.664836 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:07.674901 7040 tablet_service.cc:1467] Processing CreateTablet for tablet f209ec165a034a1682000ba382ad5edb (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.674899 6910 tablet_service.cc:1467] Processing CreateTablet for tablet f209ec165a034a1682000ba382ad5edb (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.675390 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f209ec165a034a1682000ba382ad5edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.675398 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f209ec165a034a1682000ba382ad5edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.676311 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfc52354bde44a886f6aaf3b9231f38 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.676745 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfc52354bde44a886f6aaf3b9231f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.680011 6779 tablet_service.cc:1467] Processing CreateTablet for tablet f209ec165a034a1682000ba382ad5edb (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.680413 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f209ec165a034a1682000ba382ad5edb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.684141 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.684646 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7661a6f5f34a668d5a998c7ce81276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.685575 8962 tablet_bootstrap.cc:492] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:07.687714 8962 tablet_bootstrap.cc:654] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.688206 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfc52354bde44a886f6aaf3b9231f38 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.688746 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfc52354bde44a886f6aaf3b9231f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.691439 8963 tablet_bootstrap.cc:492] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:07.692319 7169 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.692706 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7661a6f5f34a668d5a998c7ce81276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.693287 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.693691 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7661a6f5f34a668d5a998c7ce81276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.694092 8963 tablet_bootstrap.cc:654] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.695981 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfc52354bde44a886f6aaf3b9231f38 (DEFAULT_TABLE table=rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:07.696328 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfc52354bde44a886f6aaf3b9231f38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:07.705933 8965 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:07.708256 8965 tablet_bootstrap.cc:654] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.709245 8962 tablet_bootstrap.cc:492] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:07.709400 8962 ts_tablet_manager.cc:1367] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.002s
I20240503 14:13:07.710353 8962 raft_consensus.cc:348] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.710565 8962 raft_consensus.cc:374] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.710649 8962 raft_consensus.cc:729] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.710863 8962 consensus_queue.cc:260] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.711678 8962 ts_tablet_manager.cc:1398] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 14:13:07.713850 8963 tablet_bootstrap.cc:492] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:07.714011 8963 ts_tablet_manager.cc:1367] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20240503 14:13:07.714766 8963 raft_consensus.cc:348] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:07.714975 8963 raft_consensus.cc:374] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.715309 8963 raft_consensus.cc:729] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.715569 8963 consensus_queue.cc:260] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:07.716223 8963 ts_tablet_manager.cc:1398] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:07.726123 8963 tablet_bootstrap.cc:492] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:07.727298 8965 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:07.729292 8965 ts_tablet_manager.cc:1367] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240503 14:13:07.730352 8965 raft_consensus.cc:348] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.730540 8962 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:07.730646 8965 raft_consensus.cc:374] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.730758 8965 raft_consensus.cc:729] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.731012 8965 consensus_queue.cc:260] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.732815 8962 tablet_bootstrap.cc:654] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.733584 8968 tablet_bootstrap.cc:492] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:07.735229 8965 ts_tablet_manager.cc:1398] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:07.735759 8963 tablet_bootstrap.cc:654] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.737686 8968 tablet_bootstrap.cc:654] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.738659 8919 raft_consensus.cc:482] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:07.738813 8919 raft_consensus.cc:504] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:07.739848 8919 leader_election.cc:288] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:07.740157 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfc52354bde44a886f6aaf3b9231f38" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
W20240503 14:13:07.740751 6734 leader_election.cc:341] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:07.740777 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfc52354bde44a886f6aaf3b9231f38" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
W20240503 14:13:07.741822 6736 leader_election.cc:341] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:07.742059 8963 tablet_bootstrap.cc:492] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:07.742017 6736 leader_election.cc:302] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1a669e28f014fe287932acf56bebc13; no voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:07.742234 8963 ts_tablet_manager.cc:1367] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240503 14:13:07.742853 8919 raft_consensus.cc:2726] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:07.743431 8963 raft_consensus.cc:348] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.743662 8963 raft_consensus.cc:374] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.743750 8963 raft_consensus.cc:729] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.743911 8962 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:07.744146 8962 ts_tablet_manager.cc:1367] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.003s
I20240503 14:13:07.744995 8962 raft_consensus.cc:348] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.744033 8963 consensus_queue.cc:260] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.745468 8962 raft_consensus.cc:374] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.745597 8962 raft_consensus.cc:729] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.745939 8963 ts_tablet_manager.cc:1398] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:07.745904 8962 consensus_queue.cc:260] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.746726 8962 ts_tablet_manager.cc:1398] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:07.747071 8962 tablet_bootstrap.cc:492] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:07.748725 8962 tablet_bootstrap.cc:654] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.750612 8968 tablet_bootstrap.cc:492] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:07.750841 8968 ts_tablet_manager.cc:1367] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.007s
I20240503 14:13:07.751724 8968 raft_consensus.cc:348] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:07.751986 8968 raft_consensus.cc:374] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.752105 8968 raft_consensus.cc:729] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.746307 8963 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:07.752440 8968 consensus_queue.cc:260] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:07.753146 8968 ts_tablet_manager.cc:1398] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:07.755106 8968 tablet_bootstrap.cc:492] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:07.757787 8963 tablet_bootstrap.cc:654] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.761412 8968 tablet_bootstrap.cc:654] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:07.762284 8962 tablet_bootstrap.cc:492] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:07.762537 8962 ts_tablet_manager.cc:1367] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 14:13:07.763638 8962 raft_consensus.cc:348] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:07.763984 8962 raft_consensus.cc:374] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.764140 8962 raft_consensus.cc:729] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.764528 8962 consensus_queue.cc:260] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:07.765327 8962 ts_tablet_manager.cc:1398] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:07.767840 8968 tablet_bootstrap.cc:492] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:07.768023 8968 ts_tablet_manager.cc:1367] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20240503 14:13:07.770421 8968 raft_consensus.cc:348] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.770787 8968 raft_consensus.cc:374] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.770892 8968 raft_consensus.cc:729] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.771201 8968 consensus_queue.cc:260] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.773236 8968 ts_tablet_manager.cc:1398] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:07.784803 8963 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:07.785105 8963 ts_tablet_manager.cc:1367] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.021s
I20240503 14:13:07.785775 8963 raft_consensus.cc:348] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.785950 8963 raft_consensus.cc:374] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:07.786084 8963 raft_consensus.cc:729] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:07.786407 8963 consensus_queue.cc:260] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.787072 8963 ts_tablet_manager.cc:1398] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:07.797714 8701 raft_consensus.cc:482] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:07.797927 8701 raft_consensus.cc:504] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.798802 8701 leader_election.cc:288] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:07.799209 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7661a6f5f34a668d5a998c7ce81276" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:07.799417 6799 raft_consensus.cc:2454] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:07.799800 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7661a6f5f34a668d5a998c7ce81276" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:07.799856 6995 leader_election.cc:302] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:07.800067 7189 raft_consensus.cc:2454] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:07.800403 8701 raft_consensus.cc:2781] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:07.800526 8701 raft_consensus.cc:482] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:07.800606 8701 raft_consensus.cc:3037] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.803408 8701 raft_consensus.cc:504] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.804526 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7661a6f5f34a668d5a998c7ce81276" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:07.804767 8701 leader_election.cc:288] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:07.804772 7189 raft_consensus.cc:3037] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.806011 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a7661a6f5f34a668d5a998c7ce81276" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:07.806172 6799 raft_consensus.cc:3037] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.807498 7189 raft_consensus.cc:2454] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:07.808349 6799 raft_consensus.cc:2454] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:07.808689 6996 leader_election.cc:302] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:07.808969 8701 raft_consensus.cc:2781] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:07.809125 8701 raft_consensus.cc:686] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:07.809376 8701 consensus_queue.cc:237] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.812500 6655 catalog_manager.cc:5484] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:07.880749 8919 raft_consensus.cc:482] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:07.880935 8919 raft_consensus.cc:504] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.881559 8919 leader_election.cc:288] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:07.882081 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f209ec165a034a1682000ba382ad5edb" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:07.882104 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f209ec165a034a1682000ba382ad5edb" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:07.882287 7061 raft_consensus.cc:2454] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:07.882301 6931 raft_consensus.cc:2454] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:07.882774 6736 leader_election.cc:302] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:07.883070 8820 raft_consensus.cc:2781] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:07.883172 8820 raft_consensus.cc:482] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:07.883303 8820 raft_consensus.cc:3037] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.885803 8820 raft_consensus.cc:504] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.886543 8820 leader_election.cc:288] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:07.886960 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f209ec165a034a1682000ba382ad5edb" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:07.887174 7061 raft_consensus.cc:3037] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.887116 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f209ec165a034a1682000ba382ad5edb" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:07.887300 6931 raft_consensus.cc:3037] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:07.889248 6931 raft_consensus.cc:2454] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:07.889286 7061 raft_consensus.cc:2454] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:07.889690 6734 leader_election.cc:302] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:07.889993 8820 raft_consensus.cc:2781] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:07.890167 8820 raft_consensus.cc:686] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:07.890403 8820 consensus_queue.cc:237] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:07.893733 6655 catalog_manager.cc:5484] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:08.201853 8765 raft_consensus.cc:482] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:08.202032 8765 raft_consensus.cc:504] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:08.202694 8765 leader_election.cc:288] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:08.203161 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfc52354bde44a886f6aaf3b9231f38" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:08.203162 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfc52354bde44a886f6aaf3b9231f38" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:08.203421 7061 raft_consensus.cc:2454] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:08.203421 6799 raft_consensus.cc:2454] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:08.203949 6865 leader_election.cc:302] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:08.204222 8765 raft_consensus.cc:2781] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:08.204321 8765 raft_consensus.cc:482] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:08.204384 8765 raft_consensus.cc:3037] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.206140 8765 raft_consensus.cc:504] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:08.206715 8765 leader_election.cc:288] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:08.207198 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfc52354bde44a886f6aaf3b9231f38" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:08.207243 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfc52354bde44a886f6aaf3b9231f38" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:08.207386 7061 raft_consensus.cc:3037] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.207501 6799 raft_consensus.cc:3037] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:08.209309 7061 raft_consensus.cc:2454] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:08.209471 6799 raft_consensus.cc:2454] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:08.209779 6867 leader_election.cc:302] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:08.210089 8765 raft_consensus.cc:2781] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:08.210237 8765 raft_consensus.cc:686] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:08.210500 8765 consensus_queue.cc:237] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:08.215914 6655 catalog_manager.cc:5484] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:08.233995 8701 consensus_queue.cc:1035] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.248495 8900 consensus_queue.cc:1035] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.453994 8820 consensus_queue.cc:1035] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.462225 8820 consensus_queue.cc:1035] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:08.646744 8984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:08.647083 8984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:08.655149 8984 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_14'
I20240503 14:13:08.721112 8248 consensus_queue.cc:1035] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:08.737525 8765 consensus_queue.cc:1035] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 289.717 ms
time per row: 0.144859 ms
I20240503 14:13:09.105162 6657 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:09.114168 7169 tablet_service.cc:1467] Processing CreateTablet for tablet 20c4e7c22c904daebe97897320f6226e (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.114646 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c4e7c22c904daebe97897320f6226e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.115020 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 09a755b35d694a2b89ca1f58f0fa7976 (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.115516 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a755b35d694a2b89ca1f58f0fa7976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.115525 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4363edcc954769a3ab9c658e198c2d (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.115953 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4363edcc954769a3ab9c658e198c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.119279 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 09a755b35d694a2b89ca1f58f0fa7976 (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.119688 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a755b35d694a2b89ca1f58f0fa7976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.120035 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 20c4e7c22c904daebe97897320f6226e (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.120422 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c4e7c22c904daebe97897320f6226e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.124068 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4363edcc954769a3ab9c658e198c2d (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.124236 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 20c4e7c22c904daebe97897320f6226e (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.124567 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4363edcc954769a3ab9c658e198c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.124768 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c4e7c22c904daebe97897320f6226e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.128178 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 09a755b35d694a2b89ca1f58f0fa7976 (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.128711 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a755b35d694a2b89ca1f58f0fa7976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.129778 9044 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:09.131335 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 4b4363edcc954769a3ab9c658e198c2d (DEFAULT_TABLE table=rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:09.131673 9044 tablet_bootstrap.cc:654] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.131775 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4363edcc954769a3ab9c658e198c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:09.132095 9045 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:09.134294 9045 tablet_bootstrap.cc:654] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.142848 9045 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:09.143381 9045 ts_tablet_manager.cc:1367] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20240503 14:13:09.143741 9048 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:09.144276 9045 raft_consensus.cc:348] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.144526 9045 raft_consensus.cc:374] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.144662 9045 raft_consensus.cc:729] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.144934 9045 consensus_queue.cc:260] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.146487 9048 tablet_bootstrap.cc:654] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.146909 9045 ts_tablet_manager.cc:1398] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:09.149140 9044 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:09.149343 9044 ts_tablet_manager.cc:1367] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.007s
I20240503 14:13:09.150681 9044 raft_consensus.cc:348] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.151110 9044 raft_consensus.cc:374] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.151257 9044 raft_consensus.cc:729] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.152554 9048 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:09.152768 9048 ts_tablet_manager.cc:1367] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240503 14:13:09.152802 9044 consensus_queue.cc:260] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.153645 9048 raft_consensus.cc:348] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.153901 9048 raft_consensus.cc:374] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.154043 9048 raft_consensus.cc:729] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.154367 9048 consensus_queue.cc:260] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.154698 9044 ts_tablet_manager.cc:1398] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:09.154939 9048 ts_tablet_manager.cc:1398] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:09.155033 9044 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:09.158052 9048 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:09.158861 9045 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:09.159116 9044 tablet_bootstrap.cc:654] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.160742 9045 tablet_bootstrap.cc:654] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.162209 9050 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:09.163547 9048 tablet_bootstrap.cc:654] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.163930 9050 tablet_bootstrap.cc:654] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.166658 9044 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:09.166680 9045 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:09.166895 9044 ts_tablet_manager.cc:1367] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240503 14:13:09.167016 9045 ts_tablet_manager.cc:1367] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240503 14:13:09.167546 9050 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:09.167711 9050 ts_tablet_manager.cc:1367] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:13:09.168035 9044 raft_consensus.cc:348] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.168291 9044 raft_consensus.cc:374] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.168406 9044 raft_consensus.cc:729] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.168534 9050 raft_consensus.cc:348] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.168725 9050 raft_consensus.cc:374] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.168825 9050 raft_consensus.cc:729] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.168768 9044 consensus_queue.cc:260] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.169059 9050 consensus_queue.cc:260] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.169728 9050 ts_tablet_manager.cc:1398] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:09.170481 9045 raft_consensus.cc:348] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.170763 9050 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:09.170768 9045 raft_consensus.cc:374] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.170938 9045 raft_consensus.cc:729] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.171252 9045 consensus_queue.cc:260] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.172415 9050 tablet_bootstrap.cc:654] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.172430 9048 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:09.172593 9048 ts_tablet_manager.cc:1367] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20240503 14:13:09.173120 9045 ts_tablet_manager.cc:1398] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:09.173697 9044 ts_tablet_manager.cc:1398] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:09.173686 9048 raft_consensus.cc:348] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.174010 9048 raft_consensus.cc:374] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.174136 9048 raft_consensus.cc:729] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.174368 9044 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:09.174484 9048 consensus_queue.cc:260] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.175379 9048 ts_tablet_manager.cc:1398] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:09.178370 9050 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:09.178573 9050 ts_tablet_manager.cc:1367] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240503 14:13:09.179378 9050 raft_consensus.cc:348] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.179538 9044 tablet_bootstrap.cc:654] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:09.179699 8942 raft_consensus.cc:482] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:09.179750 9050 raft_consensus.cc:374] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.179831 9050 raft_consensus.cc:729] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.179860 8942 raft_consensus.cc:504] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.180133 9050 consensus_queue.cc:260] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.180821 8942 leader_election.cc:288] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:09.181200 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4363edcc954769a3ab9c658e198c2d" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:09.181425 7061 raft_consensus.cc:2454] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:09.181788 9050 ts_tablet_manager.cc:1398] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:09.182101 7127 leader_election.cc:302] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:09.182214 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4363edcc954769a3ab9c658e198c2d" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:09.182438 6929 raft_consensus.cc:2454] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:09.183166 9044 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:09.183315 9044 ts_tablet_manager.cc:1367] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:09.183559 8942 raft_consensus.cc:2781] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:09.183769 8942 raft_consensus.cc:482] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:09.183883 8942 raft_consensus.cc:3037] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.185724 9044 raft_consensus.cc:348] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.185938 9044 raft_consensus.cc:374] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:09.186061 9044 raft_consensus.cc:729] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:09.186349 9044 consensus_queue.cc:260] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.188856 9044 ts_tablet_manager.cc:1398] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:09.188942 8942 raft_consensus.cc:504] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.189656 8942 leader_election.cc:288] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:09.190405 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4363edcc954769a3ab9c658e198c2d" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:09.190621 7061 raft_consensus.cc:3037] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.191602 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b4363edcc954769a3ab9c658e198c2d" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:09.191820 6931 raft_consensus.cc:3037] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.193513 7061 raft_consensus.cc:2454] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:09.193931 7127 leader_election.cc:302] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:09.194205 9059 raft_consensus.cc:2781] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:09.194322 9059 raft_consensus.cc:686] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:09.194411 6931 raft_consensus.cc:2454] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:09.194614 9059 consensus_queue.cc:237] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.196978 8900 raft_consensus.cc:482] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:09.197144 8900 raft_consensus.cc:504] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.197841 8900 leader_election.cc:288] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:09.198280 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a755b35d694a2b89ca1f58f0fa7976" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:09.198499 7190 raft_consensus.cc:2454] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:09.198496 6657 catalog_manager.cc:5484] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:09.198951 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a755b35d694a2b89ca1f58f0fa7976" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:09.199148 6798 raft_consensus.cc:2454] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:09.199645 6995 leader_election.cc:302] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:09.199916 8900 raft_consensus.cc:2781] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:09.200022 8900 raft_consensus.cc:482] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:09.200137 8900 raft_consensus.cc:3037] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.202529 8900 raft_consensus.cc:504] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.203142 8900 leader_election.cc:288] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:09.203624 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a755b35d694a2b89ca1f58f0fa7976" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:09.203725 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a755b35d694a2b89ca1f58f0fa7976" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:09.203850 7190 raft_consensus.cc:3037] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.203941 6798 raft_consensus.cc:3037] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.205670 7190 raft_consensus.cc:2454] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:09.205873 6798 raft_consensus.cc:2454] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:09.206300 6996 leader_election.cc:302] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:09.206568 8900 raft_consensus.cc:2781] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:09.206722 8900 raft_consensus.cc:686] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:09.206975 8900 consensus_queue.cc:237] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:09.210417 6657 catalog_manager.cc:5484] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:09.302628 8248 raft_consensus.cc:482] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:09.302870 8248 raft_consensus.cc:504] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.303938 8248 leader_election.cc:288] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:09.304144 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c4e7c22c904daebe97897320f6226e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:09.304265 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c4e7c22c904daebe97897320f6226e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:09.304381 7190 raft_consensus.cc:2454] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:09.304502 6798 raft_consensus.cc:2454] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:09.304937 6866 leader_election.cc:302] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:09.305272 8248 raft_consensus.cc:2781] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:09.305392 8248 raft_consensus.cc:482] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:09.305467 8248 raft_consensus.cc:3037] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.308377 8248 raft_consensus.cc:504] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.309056 8248 leader_election.cc:288] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:09.309552 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c4e7c22c904daebe97897320f6226e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:09.310029 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20c4e7c22c904daebe97897320f6226e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:09.309798 7190 raft_consensus.cc:3037] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.310247 6798 raft_consensus.cc:3037] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:09.313087 6798 raft_consensus.cc:2454] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:09.313100 7190 raft_consensus.cc:2454] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:09.313552 6866 leader_election.cc:302] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:09.313887 8765 raft_consensus.cc:2781] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:09.314070 8765 raft_consensus.cc:686] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:09.314390 8765 consensus_queue.cc:237] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:09.318117 6657 catalog_manager.cc:5484] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:09.638370 8900 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.648304 8900 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.694113 9059 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.703186 8765 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.719879 9059 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:09.765218 8248 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:09.795172 9063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:09.795663 9063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:09.808881 9063 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_15'
INSERT report
rows total: 2000
time total: 339.984 ms
time per row: 0.169992 ms
I20240503 14:13:10.325767 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:10.335968 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 910d66fc2b224c3e9d5254d0816b31ad (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.335918 6778 tablet_service.cc:1467] Processing CreateTablet for tablet 910d66fc2b224c3e9d5254d0816b31ad (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.336287 6779 tablet_service.cc:1467] Processing CreateTablet for tablet e1a371f2fa0a4d7dbc81385e2d8df293 (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.336423 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 910d66fc2b224c3e9d5254d0816b31ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.336437 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 910d66fc2b224c3e9d5254d0816b31ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.336665 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a371f2fa0a4d7dbc81385e2d8df293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.339921 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 910d66fc2b224c3e9d5254d0816b31ad (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.340395 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 910d66fc2b224c3e9d5254d0816b31ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.341701 7041 tablet_service.cc:1467] Processing CreateTablet for tablet b98704f96ef64ccca952b039dd4f6f3b (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.342110 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98704f96ef64ccca952b039dd4f6f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.342167 6910 tablet_service.cc:1467] Processing CreateTablet for tablet e1a371f2fa0a4d7dbc81385e2d8df293 (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.342589 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a371f2fa0a4d7dbc81385e2d8df293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.344447 7170 tablet_service.cc:1467] Processing CreateTablet for tablet e1a371f2fa0a4d7dbc81385e2d8df293 (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.344983 7171 tablet_service.cc:1467] Processing CreateTablet for tablet b98704f96ef64ccca952b039dd4f6f3b (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.345206 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a371f2fa0a4d7dbc81385e2d8df293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.345429 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98704f96ef64ccca952b039dd4f6f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.346868 6780 tablet_service.cc:1467] Processing CreateTablet for tablet b98704f96ef64ccca952b039dd4f6f3b (DEFAULT_TABLE table=rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:10.347303 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b98704f96ef64ccca952b039dd4f6f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:10.354858 9129 tablet_bootstrap.cc:492] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:10.356741 9129 tablet_bootstrap.cc:654] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.359848 9127 tablet_bootstrap.cc:492] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:10.362015 9127 tablet_bootstrap.cc:654] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.363396 9128 tablet_bootstrap.cc:492] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:10.367722 9128 tablet_bootstrap.cc:654] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.369138 9129 tablet_bootstrap.cc:492] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:10.369406 9129 ts_tablet_manager.cc:1367] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.003s
I20240503 14:13:10.370630 9127 tablet_bootstrap.cc:492] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:10.370522 9129 raft_consensus.cc:348] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.370875 9127 ts_tablet_manager.cc:1367] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240503 14:13:10.370944 9129 raft_consensus.cc:374] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.371084 9129 raft_consensus.cc:729] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.371384 9129 consensus_queue.cc:260] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.372247 9129 ts_tablet_manager.cc:1398] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:10.372934 9129 tablet_bootstrap.cc:492] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:10.371817 9127 raft_consensus.cc:348] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.373284 9127 raft_consensus.cc:374] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.373431 9127 raft_consensus.cc:729] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.373847 9127 consensus_queue.cc:260] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.374542 9127 ts_tablet_manager.cc:1398] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:10.374972 9129 tablet_bootstrap.cc:654] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.376581 9127 tablet_bootstrap.cc:492] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:10.378201 9127 tablet_bootstrap.cc:654] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.378832 9109 raft_consensus.cc:482] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:10.379222 9109 raft_consensus.cc:504] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.380206 9109 leader_election.cc:288] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:10.381240 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a371f2fa0a4d7dbc81385e2d8df293" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:10.383416 9131 tablet_bootstrap.cc:492] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
W20240503 14:13:10.384696 6735 leader_election.cc:341] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:10.385402 9129 tablet_bootstrap.cc:492] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:10.385682 9129 ts_tablet_manager.cc:1367] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20240503 14:13:10.385737 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a371f2fa0a4d7dbc81385e2d8df293" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
W20240503 14:13:10.386905 6734 leader_election.cc:341] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:10.386986 9131 tablet_bootstrap.cc:654] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.386826 9129 raft_consensus.cc:348] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.387045 6734 leader_election.cc:302] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1a669e28f014fe287932acf56bebc13; no voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:10.387125 9129 raft_consensus.cc:374] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.387221 9129 raft_consensus.cc:729] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.387769 9110 raft_consensus.cc:2726] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:10.387740 9129 consensus_queue.cc:260] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.388751 9129 ts_tablet_manager.cc:1398] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:10.389205 9129 tablet_bootstrap.cc:492] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:10.391443 9129 tablet_bootstrap.cc:654] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.402467 9128 tablet_bootstrap.cc:492] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:10.402666 9128 ts_tablet_manager.cc:1367] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.005s
I20240503 14:13:10.403287 9131 tablet_bootstrap.cc:492] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:10.403458 9131 ts_tablet_manager.cc:1367] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.005s
I20240503 14:13:10.403663 9129 tablet_bootstrap.cc:492] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:10.403527 9128 raft_consensus.cc:348] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.403788 9128 raft_consensus.cc:374] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.403885 9128 raft_consensus.cc:729] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.403900 9129 ts_tablet_manager.cc:1367] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20240503 14:13:10.404136 9128 consensus_queue.cc:260] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.404618 9127 tablet_bootstrap.cc:492] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:10.404505 9131 raft_consensus.cc:348] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.404700 9128 ts_tablet_manager.cc:1398] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:10.404728 9131 raft_consensus.cc:374] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.404784 9127 ts_tablet_manager.cc:1367] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.011s
I20240503 14:13:10.404821 9131 raft_consensus.cc:729] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.405014 9128 tablet_bootstrap.cc:492] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:10.405050 9129 raft_consensus.cc:348] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.405071 9131 consensus_queue.cc:260] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.405227 9129 raft_consensus.cc:374] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.405309 9129 raft_consensus.cc:729] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.405555 9129 consensus_queue.cc:260] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.405598 9127 raft_consensus.cc:348] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.405820 9127 raft_consensus.cc:374] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.405884 9127 raft_consensus.cc:729] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.406190 9127 consensus_queue.cc:260] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.406700 9128 tablet_bootstrap.cc:654] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.406740 9131 ts_tablet_manager.cc:1398] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 14:13:10.407023 9131 tablet_bootstrap.cc:492] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:10.407224 9129 ts_tablet_manager.cc:1398] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:10.407958 9120 raft_consensus.cc:482] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:10.408085 9120 raft_consensus.cc:504] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.408824 9120 leader_election.cc:288] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:10.409873 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a371f2fa0a4d7dbc81385e2d8df293" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:10.410207 6798 raft_consensus.cc:2454] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:10.410733 9127 ts_tablet_manager.cc:1398] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:13:10.410980 9128 tablet_bootstrap.cc:492] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:10.411180 9128 ts_tablet_manager.cc:1367] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 14:13:10.411932 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a371f2fa0a4d7dbc81385e2d8df293" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:10.411967 9128 raft_consensus.cc:348] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.412139 7191 raft_consensus.cc:2454] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:10.412215 9128 raft_consensus.cc:374] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.412297 9128 raft_consensus.cc:729] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.412586 9128 consensus_queue.cc:260] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.413123 9128 ts_tablet_manager.cc:1398] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:10.413650 6865 leader_election.cc:302] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:10.414144 9120 raft_consensus.cc:2781] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:10.414254 9120 raft_consensus.cc:482] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:10.414355 9120 raft_consensus.cc:3037] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.416761 9131 tablet_bootstrap.cc:654] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:10.417181 9120 raft_consensus.cc:504] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.418084 9120 leader_election.cc:288] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:10.418267 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a371f2fa0a4d7dbc81385e2d8df293" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:10.418438 6798 raft_consensus.cc:3037] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.419536 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a371f2fa0a4d7dbc81385e2d8df293" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:10.419807 7191 raft_consensus.cc:3037] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.420315 9131 tablet_bootstrap.cc:492] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:10.420430 9131 ts_tablet_manager.cc:1367] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:10.420969 6798 raft_consensus.cc:2454] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:10.420997 9131 raft_consensus.cc:348] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.421140 9131 raft_consensus.cc:374] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:10.421185 9131 raft_consensus.cc:729] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:10.421386 9131 consensus_queue.cc:260] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.421480 6865 leader_election.cc:302] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:10.421782 8765 raft_consensus.cc:2781] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:10.421919 8765 raft_consensus.cc:686] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:10.422188 8765 consensus_queue.cc:237] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:10.422535 7191 raft_consensus.cc:2454] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:10.422607 9131 ts_tablet_manager.cc:1398] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:10.427402 6658 catalog_manager.cc:5484] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:10.555704 9109 raft_consensus.cc:482] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:10.555917 9109 raft_consensus.cc:504] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.556537 9109 leader_election.cc:288] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:10.557155 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98704f96ef64ccca952b039dd4f6f3b" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:10.557183 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98704f96ef64ccca952b039dd4f6f3b" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:10.557451 7059 raft_consensus.cc:2454] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:10.557410 7191 raft_consensus.cc:2454] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:10.557922 6735 leader_election.cc:302] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:10.558307 9109 raft_consensus.cc:2781] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:10.558409 9109 raft_consensus.cc:482] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:10.558529 9109 raft_consensus.cc:3037] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.560534 9109 raft_consensus.cc:504] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.561105 9109 leader_election.cc:288] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:10.561652 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98704f96ef64ccca952b039dd4f6f3b" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:10.561646 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b98704f96ef64ccca952b039dd4f6f3b" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:10.561890 7191 raft_consensus.cc:3037] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.561936 7058 raft_consensus.cc:3037] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.564875 7058 raft_consensus.cc:2454] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:10.564875 7191 raft_consensus.cc:2454] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:10.565392 6735 leader_election.cc:302] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:10.565665 9110 raft_consensus.cc:2781] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:10.565830 9110 raft_consensus.cc:686] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:10.566082 9110 consensus_queue.cc:237] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:10.569737 6658 catalog_manager.cc:5484] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:10.665344 8900 raft_consensus.cc:482] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:10.665517 8900 raft_consensus.cc:504] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.666347 8900 leader_election.cc:288] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:10.666797 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "910d66fc2b224c3e9d5254d0816b31ad" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:10.666848 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "910d66fc2b224c3e9d5254d0816b31ad" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:10.667074 6929 raft_consensus.cc:2454] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:10.667073 6798 raft_consensus.cc:2454] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:10.667840 6995 leader_election.cc:302] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:10.668092 8900 raft_consensus.cc:2781] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:10.668217 8900 raft_consensus.cc:482] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:10.668308 8900 raft_consensus.cc:3037] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.670617 8900 raft_consensus.cc:504] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.671216 8900 leader_election.cc:288] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:10.671550 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "910d66fc2b224c3e9d5254d0816b31ad" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:10.671619 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "910d66fc2b224c3e9d5254d0816b31ad" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:10.671790 6929 raft_consensus.cc:3037] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.671808 6798 raft_consensus.cc:3037] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:10.673592 6798 raft_consensus.cc:2454] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:10.673601 6929 raft_consensus.cc:2454] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:10.674033 6995 leader_election.cc:302] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:10.674314 8900 raft_consensus.cc:2781] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:10.674438 8900 raft_consensus.cc:686] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:10.674762 8900 consensus_queue.cc:237] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:10.678249 6658 catalog_manager.cc:5484] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:10.895493 9120 consensus_queue.cc:1035] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:10.900460 8765 consensus_queue.cc:1035] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:11.002462 9110 consensus_queue.cc:1035] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:11.008447 9110 consensus_queue.cc:1035] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:11.082365 9143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:11.082795 9143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:11.092298 9143 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
I20240503 14:13:11.128511 8900 consensus_queue.cc:1035] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:11.133147 9121 consensus_queue.cc:1035] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 257.763 ms
time per row: 0.128882 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 14:13:12.508538 6656 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:12.518180 7040 tablet_service.cc:1467] Processing CreateTablet for tablet e431272976034209b2c40cb0e460e17a (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.518239 6910 tablet_service.cc:1467] Processing CreateTablet for tablet e431272976034209b2c40cb0e460e17a (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.518677 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e431272976034209b2c40cb0e460e17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.518675 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e431272976034209b2c40cb0e460e17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.519084 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 9b154f780fc742fa9b0efd609c307f95 (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.519351 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 9b154f780fc742fa9b0efd609c307f95 (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.519454 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b154f780fc742fa9b0efd609c307f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.519748 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b154f780fc742fa9b0efd609c307f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.524878 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 9b154f780fc742fa9b0efd609c307f95 (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.525158 6909 tablet_service.cc:1467] Processing CreateTablet for tablet f9815af88a784ba7a50a43ca776ecd20 (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.525347 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b154f780fc742fa9b0efd609c307f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.525261 7039 tablet_service.cc:1467] Processing CreateTablet for tablet f9815af88a784ba7a50a43ca776ecd20 (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.525611 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9815af88a784ba7a50a43ca776ecd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.525630 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9815af88a784ba7a50a43ca776ecd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.524869 7170 tablet_service.cc:1467] Processing CreateTablet for tablet e431272976034209b2c40cb0e460e17a (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.532097 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e431272976034209b2c40cb0e460e17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.533448 6779 tablet_service.cc:1467] Processing CreateTablet for tablet f9815af88a784ba7a50a43ca776ecd20 (DEFAULT_TABLE table=rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:12.533879 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9815af88a784ba7a50a43ca776ecd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:12.545863 9211 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:12.546134 9212 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:12.547806 9212 tablet_bootstrap.cc:654] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.548139 9211 tablet_bootstrap.cc:654] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.549988 9214 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:12.551399 9211 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:12.551805 9214 tablet_bootstrap.cc:654] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.551932 9211 ts_tablet_manager.cc:1367] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:13:12.553423 9211 raft_consensus.cc:348] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.553562 9209 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:12.553746 9212 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:12.553737 9211 raft_consensus.cc:374] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.553853 9211 raft_consensus.cc:729] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.554133 9211 consensus_queue.cc:260] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.554581 9212 ts_tablet_manager.cc:1367] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240503 14:13:12.555433 9212 raft_consensus.cc:348] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.555728 9212 raft_consensus.cc:374] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.555873 9212 raft_consensus.cc:729] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.556190 9212 consensus_queue.cc:260] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.556907 9212 ts_tablet_manager.cc:1398] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:12.558418 9209 tablet_bootstrap.cc:654] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.561478 9211 ts_tablet_manager.cc:1398] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:12.564584 9214 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:12.564918 9214 ts_tablet_manager.cc:1367] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20240503 14:13:12.566138 9211 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:12.566443 9214 raft_consensus.cc:348] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.566689 9214 raft_consensus.cc:374] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.566825 9214 raft_consensus.cc:729] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.567292 9214 consensus_queue.cc:260] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.568244 9211 tablet_bootstrap.cc:654] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.569530 9214 ts_tablet_manager.cc:1398] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:12.569847 9214 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:12.570464 9209 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:12.570708 9209 ts_tablet_manager.cc:1367] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20240503 14:13:12.571668 9209 raft_consensus.cc:348] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.571923 9209 raft_consensus.cc:374] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.572050 9209 raft_consensus.cc:729] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.572083 9214 tablet_bootstrap.cc:654] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.572319 9209 consensus_queue.cc:260] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.572943 9209 ts_tablet_manager.cc:1398] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:12.573155 9211 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:12.573340 9211 ts_tablet_manager.cc:1367] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:12.574129 9209 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:12.574132 9211 raft_consensus.cc:348] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.574357 9211 raft_consensus.cc:374] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.574424 9211 raft_consensus.cc:729] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.574656 9211 consensus_queue.cc:260] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.575359 9211 ts_tablet_manager.cc:1398] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:12.575985 9211 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:12.577910 9209 tablet_bootstrap.cc:654] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.581435 9211 tablet_bootstrap.cc:654] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.593140 9211 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:12.593377 9211 ts_tablet_manager.cc:1367] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.009s
I20240503 14:13:12.593442 9209 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:12.593672 9209 ts_tablet_manager.cc:1367] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.010s
I20240503 14:13:12.594082 9211 raft_consensus.cc:348] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.594251 9211 raft_consensus.cc:374] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.594513 9211 raft_consensus.cc:729] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.594843 9211 consensus_queue.cc:260] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.595638 9211 ts_tablet_manager.cc:1398] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:12.596230 9214 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:12.594681 9209 raft_consensus.cc:348] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.596452 9214 ts_tablet_manager.cc:1367] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.006s
I20240503 14:13:12.596452 9209 raft_consensus.cc:374] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.596539 9209 raft_consensus.cc:729] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.596791 9209 consensus_queue.cc:260] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.597345 9209 ts_tablet_manager.cc:1398] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:12.597357 9214 raft_consensus.cc:348] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.597642 9214 raft_consensus.cc:374] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.597762 9214 raft_consensus.cc:729] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.598052 9214 consensus_queue.cc:260] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.598881 9209 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:12.599877 9214 ts_tablet_manager.cc:1398] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:12.600697 9209 tablet_bootstrap.cc:654] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:12.607870 9209 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:12.608044 9209 ts_tablet_manager.cc:1367] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:13:12.608777 9209 raft_consensus.cc:348] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.609042 9209 raft_consensus.cc:374] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:12.609171 9209 raft_consensus.cc:729] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:12.609378 9209 consensus_queue.cc:260] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.609845 9209 ts_tablet_manager.cc:1398] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:12.632963 9210 raft_consensus.cc:482] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:12.633162 9210 raft_consensus.cc:504] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.633807 9210 leader_election.cc:288] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:12.634224 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:12.634332 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:12.634477 7189 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:12.634594 7058 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:12.635005 6867 leader_election.cc:302] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:12.635320 8765 raft_consensus.cc:2781] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:12.635433 8765 raft_consensus.cc:482] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:12.635541 8765 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.637761 8765 raft_consensus.cc:504] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.638365 8765 leader_election.cc:288] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:12.638667 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:12.638890 7191 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.639110 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:12.639320 7058 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.640967 7191 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:12.641381 6866 leader_election.cc:302] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:12.641448 7058 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:12.641654 8765 raft_consensus.cc:2781] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:12.641868 8765 raft_consensus.cc:686] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:12.642165 8765 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:12.645905 6656 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:12.774112 8765 raft_consensus.cc:482] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:12.774308 8765 raft_consensus.cc:504] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.775061 8765 leader_election.cc:288] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:12.775513 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e431272976034209b2c40cb0e460e17a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:12.775565 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e431272976034209b2c40cb0e460e17a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:12.775842 7189 raft_consensus.cc:2454] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:12.775844 7058 raft_consensus.cc:2454] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:12.776235 6866 leader_election.cc:302] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:12.776496 8765 raft_consensus.cc:2781] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:12.776611 8765 raft_consensus.cc:482] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:12.776691 8765 raft_consensus.cc:3037] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.778533 8765 raft_consensus.cc:504] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.779072 8765 leader_election.cc:288] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:12.779559 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e431272976034209b2c40cb0e460e17a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:12.779613 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e431272976034209b2c40cb0e460e17a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:12.779793 7058 raft_consensus.cc:3037] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.779803 7189 raft_consensus.cc:3037] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.781689 7189 raft_consensus.cc:2454] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:12.781904 7058 raft_consensus.cc:2454] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:12.782099 6866 leader_election.cc:302] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:12.782383 8765 raft_consensus.cc:2781] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:12.782563 8765 raft_consensus.cc:686] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:12.783278 8765 consensus_queue.cc:237] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:12.787112 6656 catalog_manager.cc:5484] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:12.848613 8765 raft_consensus.cc:482] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:12.848865 8765 raft_consensus.cc:504] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.849574 8765 leader_election.cc:288] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:12.849931 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9815af88a784ba7a50a43ca776ecd20" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:12.850059 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9815af88a784ba7a50a43ca776ecd20" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:12.850158 7058 raft_consensus.cc:2454] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:12.850275 6798 raft_consensus.cc:2454] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:12.850564 6867 leader_election.cc:302] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:12.850876 8765 raft_consensus.cc:2781] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:12.850956 8765 raft_consensus.cc:482] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:12.851054 8765 raft_consensus.cc:3037] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.853044 8765 raft_consensus.cc:504] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.853648 8765 leader_election.cc:288] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:12.854100 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9815af88a784ba7a50a43ca776ecd20" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:12.854130 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9815af88a784ba7a50a43ca776ecd20" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:12.854283 7058 raft_consensus.cc:3037] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.854306 6798 raft_consensus.cc:3037] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:12.856657 6798 raft_consensus.cc:2454] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:12.856706 7058 raft_consensus.cc:2454] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:12.857071 6865 leader_election.cc:302] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:12.857362 8765 raft_consensus.cc:2781] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:12.857514 8765 raft_consensus.cc:686] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:12.857743 8765 consensus_queue.cc:237] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:12.860953 6656 catalog_manager.cc:5484] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:13.174827 8765 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.184162 9230 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.208343 9230 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.214907 9210 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.304286 8765 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:13.316568 8765 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:13.323803 9227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:13.324221 9227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:13.333667 9227 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
INSERT report
rows total: 2000
time total: 227.668 ms
time per row: 0.113834 ms
I20240503 14:13:13.731164 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:13.741824 7171 tablet_service.cc:1467] Processing CreateTablet for tablet fdd011ee413741b0be51e718992c1f4a (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.742274 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd011ee413741b0be51e718992c1f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.743046 6779 tablet_service.cc:1467] Processing CreateTablet for tablet fdd011ee413741b0be51e718992c1f4a (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.743263 6778 tablet_service.cc:1467] Processing CreateTablet for tablet c18ac0e1d6c94714a1021a9888319f3c (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.743463 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd011ee413741b0be51e718992c1f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.744072 6778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18ac0e1d6c94714a1021a9888319f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.744015 6911 tablet_service.cc:1467] Processing CreateTablet for tablet fdd011ee413741b0be51e718992c1f4a (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.744307 7040 tablet_service.cc:1467] Processing CreateTablet for tablet e634f4f40a8c4bc1925498d384a30c01 (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.744417 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdd011ee413741b0be51e718992c1f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.744719 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e634f4f40a8c4bc1925498d384a30c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.746920 6910 tablet_service.cc:1467] Processing CreateTablet for tablet c18ac0e1d6c94714a1021a9888319f3c (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.747318 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18ac0e1d6c94714a1021a9888319f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.756006 6780 tablet_service.cc:1467] Processing CreateTablet for tablet e634f4f40a8c4bc1925498d384a30c01 (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.756516 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e634f4f40a8c4bc1925498d384a30c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.756953 9291 tablet_bootstrap.cc:492] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:13.757375 9292 tablet_bootstrap.cc:492] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:13.758965 9291 tablet_bootstrap.cc:654] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.761137 9292 tablet_bootstrap.cc:654] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.761245 7170 tablet_service.cc:1467] Processing CreateTablet for tablet e634f4f40a8c4bc1925498d384a30c01 (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.761664 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e634f4f40a8c4bc1925498d384a30c01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.768057 7169 tablet_service.cc:1467] Processing CreateTablet for tablet c18ac0e1d6c94714a1021a9888319f3c (DEFAULT_TABLE table=rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:13.768314 9291 tablet_bootstrap.cc:492] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:13.768476 7169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c18ac0e1d6c94714a1021a9888319f3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:13.768507 9291 ts_tablet_manager.cc:1367] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240503 14:13:13.769604 9291 raft_consensus.cc:348] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.770009 9291 raft_consensus.cc:374] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.770125 9291 raft_consensus.cc:729] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.770438 9291 consensus_queue.cc:260] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.771199 9291 ts_tablet_manager.cc:1398] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 14:13:13.789040 9292 tablet_bootstrap.cc:492] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:13.789214 9292 ts_tablet_manager.cc:1367] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.024s
I20240503 14:13:13.789878 9296 tablet_bootstrap.cc:492] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:13.790026 9292 raft_consensus.cc:348] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.790235 9292 raft_consensus.cc:374] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.790362 9292 raft_consensus.cc:729] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.790596 9292 consensus_queue.cc:260] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.791337 9292 ts_tablet_manager.cc:1398] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:13.791792 9292 tablet_bootstrap.cc:492] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:13.792726 9295 tablet_bootstrap.cc:492] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:13.793431 9292 tablet_bootstrap.cc:654] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.794433 9296 tablet_bootstrap.cc:654] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.796605 9295 tablet_bootstrap.cc:654] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.801087 9296 tablet_bootstrap.cc:492] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:13.801327 9296 ts_tablet_manager.cc:1367] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:13:13.802104 9296 raft_consensus.cc:348] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.802428 9296 raft_consensus.cc:374] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.802568 9296 raft_consensus.cc:729] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.802897 9296 consensus_queue.cc:260] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.803479 9296 ts_tablet_manager.cc:1398] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:13.803943 9296 tablet_bootstrap.cc:492] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:13.805668 9296 tablet_bootstrap.cc:654] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.807364 9295 tablet_bootstrap.cc:492] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:13.807583 9295 ts_tablet_manager.cc:1367] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240503 14:13:13.808456 9295 raft_consensus.cc:348] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.808748 9295 raft_consensus.cc:374] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.808899 9295 raft_consensus.cc:729] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.809176 9295 consensus_queue.cc:260] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.810860 9295 ts_tablet_manager.cc:1398] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:13.811197 9295 tablet_bootstrap.cc:492] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:13.814481 9295 tablet_bootstrap.cc:654] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.814606 9292 tablet_bootstrap.cc:492] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:13.814774 9292 ts_tablet_manager.cc:1367] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.016s
I20240503 14:13:13.815549 9292 raft_consensus.cc:348] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.815798 9292 raft_consensus.cc:374] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.815865 9292 raft_consensus.cc:729] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.816090 9292 consensus_queue.cc:260] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.819149 9292 ts_tablet_manager.cc:1398] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:13.819479 9292 tablet_bootstrap.cc:492] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:13.822806 9292 tablet_bootstrap.cc:654] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.823312 9296 tablet_bootstrap.cc:492] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:13.823555 9296 ts_tablet_manager.cc:1367] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240503 14:13:13.824388 9296 raft_consensus.cc:348] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.824579 9296 raft_consensus.cc:374] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.824661 9296 raft_consensus.cc:729] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.824976 9296 consensus_queue.cc:260] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.826710 9296 ts_tablet_manager.cc:1398] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:13.828374 9295 tablet_bootstrap.cc:492] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:13.828776 9295 ts_tablet_manager.cc:1367] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 14:13:13.832405 9203 raft_consensus.cc:482] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:13.832749 9292 tablet_bootstrap.cc:492] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:13.832944 9292 ts_tablet_manager.cc:1367] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240503 14:13:13.832751 9203 raft_consensus.cc:504] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.833618 9203 leader_election.cc:288] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:13.833832 9292 raft_consensus.cc:348] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.834026 9292 raft_consensus.cc:374] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.834115 9292 raft_consensus.cc:729] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.834151 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18ac0e1d6c94714a1021a9888319f3c" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:13.834331 9292 consensus_queue.cc:260] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.834885 9292 ts_tablet_manager.cc:1398] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:13.836537 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18ac0e1d6c94714a1021a9888319f3c" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:13.836786 6931 raft_consensus.cc:2454] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:13.838248 6734 leader_election.cc:302] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:13.838546 9237 raft_consensus.cc:2781] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:13.838639 9237 raft_consensus.cc:482] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:13.838725 9237 raft_consensus.cc:3037] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.829630 9295 raft_consensus.cc:348] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.839068 9295 raft_consensus.cc:374] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.839167 9295 raft_consensus.cc:729] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.839412 9295 consensus_queue.cc:260] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.839679 7189 raft_consensus.cc:2454] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:13.840189 9295 ts_tablet_manager.cc:1398] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 14:13:13.841009 9295 tablet_bootstrap.cc:492] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:13.842761 9295 tablet_bootstrap.cc:654] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:13.846035 9237 raft_consensus.cc:504] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.848121 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18ac0e1d6c94714a1021a9888319f3c" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:13.848333 7189 raft_consensus.cc:3037] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.850355 7189 raft_consensus.cc:2454] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:13.850778 6735 leader_election.cc:302] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:13.851286 9203 raft_consensus.cc:2781] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:13.852160 9237 leader_election.cc:288] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:13.852758 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c18ac0e1d6c94714a1021a9888319f3c" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:13.853031 6931 raft_consensus.cc:3037] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.857003 6931 raft_consensus.cc:2454] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:13.859941 9203 raft_consensus.cc:686] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:13.860275 9203 consensus_queue.cc:237] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:13.861173 9295 tablet_bootstrap.cc:492] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:13.861328 9295 ts_tablet_manager.cc:1367] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.014s
I20240503 14:13:13.861943 9295 raft_consensus.cc:348] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.862112 9295 raft_consensus.cc:374] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:13.862200 9295 raft_consensus.cc:729] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:13.862411 9295 consensus_queue.cc:260] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.862908 9295 ts_tablet_manager.cc:1398] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:13.864059 6654 catalog_manager.cc:5484] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:13.913031 9230 raft_consensus.cc:482] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:13.913275 9230 raft_consensus.cc:504] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.913993 9230 leader_election.cc:288] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:13.914546 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd011ee413741b0be51e718992c1f4a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:13.914566 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd011ee413741b0be51e718992c1f4a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:13.914785 6800 raft_consensus.cc:2454] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:13.914803 7189 raft_consensus.cc:2454] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:13.915266 6865 leader_election.cc:302] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:13.915563 9230 raft_consensus.cc:2781] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:13.915704 9230 raft_consensus.cc:482] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:13.915792 9230 raft_consensus.cc:3037] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.917626 9230 raft_consensus.cc:504] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.918192 9230 leader_election.cc:288] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:13.918594 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd011ee413741b0be51e718992c1f4a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:13.918643 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdd011ee413741b0be51e718992c1f4a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:13.918788 6800 raft_consensus.cc:3037] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.918877 7189 raft_consensus.cc:3037] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.920815 6800 raft_consensus.cc:2454] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:13.921005 7189 raft_consensus.cc:2454] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:13.921238 6865 leader_election.cc:302] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:13.921579 9230 raft_consensus.cc:2781] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:13.921732 9230 raft_consensus.cc:686] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:13.922046 9230 consensus_queue.cc:237] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.926088 6657 catalog_manager.cc:5484] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:13.967434 9193 raft_consensus.cc:482] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:13.967681 9193 raft_consensus.cc:504] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.968353 9193 leader_election.cc:288] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:13.968808 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e634f4f40a8c4bc1925498d384a30c01" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:13.968874 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e634f4f40a8c4bc1925498d384a30c01" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:13.969043 6800 raft_consensus.cc:2454] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:13.969105 7189 raft_consensus.cc:2454] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:13.969448 6995 leader_election.cc:302] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:13.969707 9193 raft_consensus.cc:2781] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:13.969830 9193 raft_consensus.cc:482] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:13.969926 9193 raft_consensus.cc:3037] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.971868 9193 raft_consensus.cc:504] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.972394 9193 leader_election.cc:288] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:13.972802 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e634f4f40a8c4bc1925498d384a30c01" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:13.972870 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e634f4f40a8c4bc1925498d384a30c01" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:13.972965 6800 raft_consensus.cc:3037] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.973085 7189 raft_consensus.cc:3037] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:13.974845 6800 raft_consensus.cc:2454] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:13.974891 7189 raft_consensus.cc:2454] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:13.975275 6995 leader_election.cc:302] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:13.975536 9193 raft_consensus.cc:2781] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:13.975704 9193 raft_consensus.cc:686] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:13.975963 9193 consensus_queue.cc:237] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:13.979148 6657 catalog_manager.cc:5484] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:14.317551 9203 consensus_queue.cc:1035] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:14.326754 9304 consensus_queue.cc:1035] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:14.348598 9230 consensus_queue.cc:1035] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:14.354588 9276 consensus_queue.cc:1035] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:14.391212 9308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:14.391542 9308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:14.399443 9308 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240503 14:13:14.434513 8900 consensus_queue.cc:1035] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:14.441022 8900 consensus_queue.cc:1035] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 293 ms
time per row: 0.1465 ms
I20240503 14:13:14.865854 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:14.880980 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 21e68127f4f94413b1be984a87bec6a8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.881379 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e68127f4f94413b1be984a87bec6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.881958 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 21e68127f4f94413b1be984a87bec6a8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.882378 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e68127f4f94413b1be984a87bec6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.882278 7171 tablet_service.cc:1467] Processing CreateTablet for tablet b8014635e8294062b1c1985a100307ff (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.882895 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8014635e8294062b1c1985a100307ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.884479 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 3160f1a600604f258f76614f13429563 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.884864 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3160f1a600604f258f76614f13429563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.884774 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 21e68127f4f94413b1be984a87bec6a8 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.884905 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 3160f1a600604f258f76614f13429563 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.885205 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e68127f4f94413b1be984a87bec6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.885334 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3160f1a600604f258f76614f13429563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.888355 6910 tablet_service.cc:1467] Processing CreateTablet for tablet b8014635e8294062b1c1985a100307ff (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.888753 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8014635e8294062b1c1985a100307ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.891930 7039 tablet_service.cc:1467] Processing CreateTablet for tablet b8014635e8294062b1c1985a100307ff (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.892344 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8014635e8294062b1c1985a100307ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.895766 7041 tablet_service.cc:1467] Processing CreateTablet for tablet 3160f1a600604f258f76614f13429563 (DEFAULT_TABLE table=rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:14.896162 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3160f1a600604f258f76614f13429563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:14.908411 9375 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:14.913112 9375 tablet_bootstrap.cc:654] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.913184 9376 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:14.915778 9374 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:14.915946 9376 tablet_bootstrap.cc:654] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.917273 9374 tablet_bootstrap.cc:654] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.917537 9373 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:14.918848 9375 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:14.919162 9375 ts_tablet_manager.cc:1367] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240503 14:13:14.920598 9375 raft_consensus.cc:348] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:14.920893 9375 raft_consensus.cc:374] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.920995 9373 tablet_bootstrap.cc:654] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.921068 9375 raft_consensus.cc:729] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.921358 9375 consensus_queue.cc:260] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:14.922037 9375 ts_tablet_manager.cc:1398] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:14.922689 9375 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:14.925828 9375 tablet_bootstrap.cc:654] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.928920 9376 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:14.928987 9374 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:14.929127 9376 ts_tablet_manager.cc:1367] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240503 14:13:14.929214 9374 ts_tablet_manager.cc:1367] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 14:13:14.930048 9374 raft_consensus.cc:348] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:14.930182 9376 raft_consensus.cc:348] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.930461 9374 raft_consensus.cc:374] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.930470 9376 raft_consensus.cc:374] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.930543 9374 raft_consensus.cc:729] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.930570 9376 raft_consensus.cc:729] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.930810 9374 consensus_queue.cc:260] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:14.931460 9374 ts_tablet_manager.cc:1398] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:14.931324 9376 consensus_queue.cc:260] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.932336 9376 ts_tablet_manager.cc:1398] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:14.932801 9376 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:14.934523 9374 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:14.936095 9376 tablet_bootstrap.cc:654] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.937508 9375 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:14.937655 9374 tablet_bootstrap.cc:654] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.937813 9375 ts_tablet_manager.cc:1367] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20240503 14:13:14.941347 9374 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:14.941558 9374 ts_tablet_manager.cc:1367] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20240503 14:13:14.941875 9375 raft_consensus.cc:348] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:14.942239 9375 raft_consensus.cc:374] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.942375 9375 raft_consensus.cc:729] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.943020 9374 raft_consensus.cc:348] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.943117 9375 consensus_queue.cc:260] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:14.943316 9374 raft_consensus.cc:374] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.943536 9374 raft_consensus.cc:729] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.943954 9375 ts_tablet_manager.cc:1398] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:14.943989 9374 consensus_queue.cc:260] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.944581 9374 ts_tablet_manager.cc:1398] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:14.948480 9376 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:14.948704 9376 ts_tablet_manager.cc:1367] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.007s
I20240503 14:13:14.949052 9373 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:14.949196 9373 ts_tablet_manager.cc:1367] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.000s
I20240503 14:13:14.949532 9376 raft_consensus.cc:348] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:14.949851 9376 raft_consensus.cc:374] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.949978 9376 raft_consensus.cc:729] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.949959 9373 raft_consensus.cc:348] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.950294 9376 consensus_queue.cc:260] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:14.950590 9373 raft_consensus.cc:374] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.950768 9373 raft_consensus.cc:729] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.951076 9373 consensus_queue.cc:260] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.951447 9224 raft_consensus.cc:482] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:14.951567 9224 raft_consensus.cc:504] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.952522 9224 leader_election.cc:288] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:14.952863 9376 ts_tablet_manager.cc:1398] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:14.952996 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8014635e8294062b1c1985a100307ff" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:14.953243 7059 raft_consensus.cc:2454] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:14.953249 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8014635e8294062b1c1985a100307ff" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:14.953377 9376 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:14.953469 6931 raft_consensus.cc:2454] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:14.953722 7127 leader_election.cc:302] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:14.954002 9224 raft_consensus.cc:2781] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:14.954098 9224 raft_consensus.cc:482] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:14.954159 9224 raft_consensus.cc:3037] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.957132 9224 raft_consensus.cc:504] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.958505 9224 leader_election.cc:288] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:14.958885 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8014635e8294062b1c1985a100307ff" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:14.959012 9373 ts_tablet_manager.cc:1398] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:13:14.959097 6931 raft_consensus.cc:3037] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.959353 9373 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:14.960220 9376 tablet_bootstrap.cc:654] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.961985 6931 raft_consensus.cc:2454] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:14.962252 9376 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:14.962400 9376 ts_tablet_manager.cc:1367] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:13:14.962397 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8014635e8294062b1c1985a100307ff" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:14.962416 7125 leader_election.cc:302] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:14.962657 7059 raft_consensus.cc:3037] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:14.962692 9224 raft_consensus.cc:2781] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:14.962819 9224 raft_consensus.cc:686] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:14.963053 9224 consensus_queue.cc:237] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:14.963286 9376 raft_consensus.cc:348] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:14.963486 9376 raft_consensus.cc:374] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.963579 9376 raft_consensus.cc:729] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.963824 9376 consensus_queue.cc:260] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:14.965366 9373 tablet_bootstrap.cc:654] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:14.965986 9376 ts_tablet_manager.cc:1398] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:14.966693 7059 raft_consensus.cc:2454] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:14.968263 9373 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:14.968493 9373 ts_tablet_manager.cc:1367] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:14.969512 9373 raft_consensus.cc:348] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:14.969779 9373 raft_consensus.cc:374] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:14.969906 9373 raft_consensus.cc:729] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:14.970146 9373 consensus_queue.cc:260] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:14.970665 9373 ts_tablet_manager.cc:1398] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:14.971771 6652 catalog_manager.cc:5484] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:15.112393 9193 raft_consensus.cc:482] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:15.112648 9193 raft_consensus.cc:504] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:15.113337 9193 leader_election.cc:288] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:15.113859 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e68127f4f94413b1be984a87bec6a8" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:15.113871 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e68127f4f94413b1be984a87bec6a8" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:15.114154 6800 raft_consensus.cc:2454] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:15.114241 6931 raft_consensus.cc:2454] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:15.114653 6995 leader_election.cc:302] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:15.114943 9193 raft_consensus.cc:2781] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:15.115124 9193 raft_consensus.cc:482] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:15.115265 9193 raft_consensus.cc:3037] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:15.117551 9193 raft_consensus.cc:504] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:15.118204 9193 leader_election.cc:288] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:15.118759 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e68127f4f94413b1be984a87bec6a8" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:15.118706 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e68127f4f94413b1be984a87bec6a8" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:15.119010 6931 raft_consensus.cc:3037] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:15.119031 6800 raft_consensus.cc:3037] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:15.122052 6800 raft_consensus.cc:2454] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:15.122058 6931 raft_consensus.cc:2454] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:15.122578 6995 leader_election.cc:302] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:15.122907 9193 raft_consensus.cc:2781] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:15.123095 9193 raft_consensus.cc:686] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:15.123389 9193 consensus_queue.cc:237] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:15.127422 6652 catalog_manager.cc:5484] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:15.138459 9193 raft_consensus.cc:482] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:15.138651 9193 raft_consensus.cc:504] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:15.139381 9193 leader_election.cc:288] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:15.139889 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3160f1a600604f258f76614f13429563" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:15.140033 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3160f1a600604f258f76614f13429563" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:15.140101 7191 raft_consensus.cc:2454] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:15.140295 6800 raft_consensus.cc:2454] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:15.140609 6996 leader_election.cc:302] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:15.140977 9193 raft_consensus.cc:2781] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:15.141132 9193 raft_consensus.cc:482] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:15.141232 9193 raft_consensus.cc:3037] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:15.143105 9193 raft_consensus.cc:504] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:15.143790 9193 leader_election.cc:288] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:15.144326 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3160f1a600604f258f76614f13429563" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:15.144351 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3160f1a600604f258f76614f13429563" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:15.144519 7191 raft_consensus.cc:3037] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:15.144554 6800 raft_consensus.cc:3037] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:15.146593 7191 raft_consensus.cc:2454] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:15.147109 6800 raft_consensus.cc:2454] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:15.147536 6995 leader_election.cc:302] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:15.148026 9193 raft_consensus.cc:2781] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:15.148162 9193 raft_consensus.cc:686] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:15.148454 9193 consensus_queue.cc:237] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:15.151863 6652 catalog_manager.cc:5484] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:15.475167 9287 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:15.482882 9287 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:15.514761 9193 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:15.519794 9341 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:15.607088 9389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:15.607460 9389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:15.615191 9389 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 14:13:15.720853 9396 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:15.726419 9341 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 310.251 ms
time per row: 0.155125 ms
I20240503 14:13:16.080157 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.5:36609
--local_ip_for_outbound_sockets=127.0.186.5
--tserver_master_addrs=127.0.186.62:45803
--webserver_port=45981
--webserver_interface=127.0.186.5
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 14:13:16.459725 9449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:16.460055 9449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:16.460188 9449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:16.468741 9449 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:16.468910 9449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:16.469062 9449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.5
I20240503 14:13:16.479547 9449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.5:36609
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.186.5
--webserver_port=45981
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.5
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:16.480991 9449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:16.482913 9449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:16.496999 9458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:16.497560 9455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:16.498219 9456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:16.498811 9449 server_base.cc:1034] running on GCE node
I20240503 14:13:16.499614 9449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:16.500715 9449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:16.501950 9449 hybrid_clock.cc:648] HybridClock initialized: now 1714745596501864 us; error 121 us; skew 500 ppm
I20240503 14:13:16.502314 9449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:16.504753 9449 webserver.cc:460] Webserver started at http://127.0.186.5:45981/ using document root <none> and password file <none>
I20240503 14:13:16.505540 9449 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:16.505653 9449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:16.511927 9449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 14:13:16.515388 9464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:16.516867 9449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:16.517026 9449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5b7e36bf354f480c90859cb8867b59cf"
format_stamp: "Formatted at 2024-05-03 14:12:44 on dist-test-slave-mh2c"
I20240503 14:13:16.517880 9449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:16.542721 9449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:16.543700 9449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:16.544018 9449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:16.545112 9449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:16.547447 9449 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:16.547537 9449 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:16.547706 9449 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:16.547758 9449 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:16.597106 9449 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.5:36609
I20240503 14:13:16.597182 9577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.5:36609 every 8 connection(s)
I20240503 14:13:16.598990 9449 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 14:13:16.605067 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 9449
I20240503 14:13:16.605683 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.6:33971
--local_ip_for_outbound_sockets=127.0.186.6
--tserver_master_addrs=127.0.186.62:45803
--webserver_port=46767
--webserver_interface=127.0.186.6
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:16.613441 9578 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:13:16.613821 9578 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:16.614861 9578 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:13:16.616515 6650 ts_manager.cc:194] Re-registered known tserver with Master: 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:16.617358 6650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.5:50669
W20240503 14:13:16.985345 9582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:16.985666 9582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:16.985810 9582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:16.994045 9582 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:16.994215 9582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:16.994375 9582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.6
I20240503 14:13:17.005997 9582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.6:33971
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.0.186.6
--webserver_port=46767
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.6
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:17.007761 9582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:17.009651 9582 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:17.023829 9588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:17.024566 9589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:17.025077 9591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:17.026068 9582 server_base.cc:1034] running on GCE node
I20240503 14:13:17.026844 9582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:17.027951 9582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:17.029261 9582 hybrid_clock.cc:648] HybridClock initialized: now 1714745597029181 us; error 72 us; skew 500 ppm
I20240503 14:13:17.029629 9582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:17.032136 9582 webserver.cc:460] Webserver started at http://127.0.186.6:46767/ using document root <none> and password file <none>
I20240503 14:13:17.032919 9582 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:17.033064 9582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:17.038813 9582 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 14:13:17.042019 9597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:17.043558 9582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:17.043784 9582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "f03dc37fe3234033bc665bed4f1b51f5"
format_stamp: "Formatted at 2024-05-03 14:12:44 on dist-test-slave-mh2c"
I20240503 14:13:17.044694 9582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:17.068745 9582 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:17.069653 9582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:17.069964 9582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:17.071151 9582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:17.073206 9582 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:17.073313 9582 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:17.073496 9582 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:17.073570 9582 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:17.117172 9582 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.6:33971
I20240503 14:13:17.117200 9709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.6:33971 every 8 connection(s)
I20240503 14:13:17.119352 9582 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 14:13:17.121752 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 9582
I20240503 14:13:17.122552 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.7:42783
--local_ip_for_outbound_sockets=127.0.186.7
--tserver_master_addrs=127.0.186.62:45803
--webserver_port=33785
--webserver_interface=127.0.186.7
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:17.139886 9710 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:13:17.140221 9710 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:17.141359 9710 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:13:17.143072 6650 ts_manager.cc:194] Re-registered known tserver with Master: f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:17.143913 6650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.6:42117
W20240503 14:13:17.499787 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:17.500128 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:17.500258 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:17.508841 9714 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:17.508981 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:17.509135 9714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.7
I20240503 14:13:17.519488 9714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.7:42783
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.0.186.7
--webserver_port=33785
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.7
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:17.521015 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:17.523166 9714 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:17.537911 9723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:17.538241 9725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:17.538851 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:17.539557 9714 server_base.cc:1034] running on GCE node
I20240503 14:13:17.540323 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:17.541508 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:17.542754 9714 hybrid_clock.cc:648] HybridClock initialized: now 1714745597542712 us; error 40 us; skew 500 ppm
I20240503 14:13:17.543123 9714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:17.545792 9714 webserver.cc:460] Webserver started at http://127.0.186.7:33785/ using document root <none> and password file <none>
I20240503 14:13:17.546607 9714 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:17.546747 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:17.552917 9714 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:13:17.556329 9731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:17.558436 9714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:13:17.558728 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "3efdbd3506274dba846ae40404f0572c"
format_stamp: "Formatted at 2024-05-03 14:12:45 on dist-test-slave-mh2c"
I20240503 14:13:17.560045 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:17.582800 9714 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:17.583810 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:17.584133 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:17.585227 9714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:17.586988 9714 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:17.587081 9714 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:17.587205 9714 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:17.587253 9714 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:17.620602 9578 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
I20240503 14:13:17.627049 9714 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.7:42783
I20240503 14:13:17.627051 9843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.7:42783 every 8 connection(s)
I20240503 14:13:17.628749 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 14:13:17.630291 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 9714
I20240503 14:13:17.631047 744 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
/tmp/dist-test-tasknKJaW5/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.186.8:36653
--local_ip_for_outbound_sockets=127.0.186.8
--tserver_master_addrs=127.0.186.62:45803
--webserver_port=33785
--webserver_interface=127.0.186.8
--builtin_ntp_servers=127.0.186.20:43165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 14:13:17.644398 9844 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:13:17.644726 9844 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:17.645593 9844 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:13:17.647368 6650 ts_manager.cc:194] Re-registered known tserver with Master: 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:17.648603 6650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.7:50973
W20240503 14:13:18.022838 9847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 14:13:18.023177 9847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 14:13:18.023321 9847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 14:13:18.031638 9847 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 14:13:18.031793 9847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 14:13:18.031970 9847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.186.8
I20240503 14:13:18.042383 9847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.186.20:43165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.186.8:36653
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.0.186.8
--webserver_port=33785
--tserver_master_addrs=127.0.186.62:45803
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.186.8
--log_dir=/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 13:43:24 UTC on efd1337a2aa3
build id 107
ASAN enabled
I20240503 14:13:18.044070 9847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 14:13:18.045835 9847 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 14:13:18.061087 9853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:18.062418 9852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 14:13:18.062887 9855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 14:13:18.063823 9847 server_base.cc:1034] running on GCE node
I20240503 14:13:18.064481 9847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 14:13:18.065593 9847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 14:13:18.066816 9847 hybrid_clock.cc:648] HybridClock initialized: now 1714745598066747 us; error 77 us; skew 500 ppm
I20240503 14:13:18.067188 9847 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 14:13:18.069883 9847 webserver.cc:460] Webserver started at http://127.0.186.8:33785/ using document root <none> and password file <none>
I20240503 14:13:18.070871 9847 fs_manager.cc:359] Metadata directory not provided
I20240503 14:13:18.071038 9847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 14:13:18.076884 9847 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20240503 14:13:18.079885 9861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:18.081661 9847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 14:13:18.081952 9847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "ea801601713a4d55aee0de9e3473e0bc"
format_stamp: "Formatted at 2024-05-03 14:12:45 on dist-test-slave-mh2c"
I20240503 14:13:18.083163 9847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 14:13:18.112994 9847 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 14:13:18.114106 9847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 14:13:18.114364 9847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 14:13:18.115295 9847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 14:13:18.117391 9847 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 14:13:18.117487 9847 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:18.117602 9847 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 14:13:18.117664 9847 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 14:13:18.147348 9710 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
I20240503 14:13:18.164577 9847 rpc_server.cc:307] RPC server started. Bound to: 127.0.186.8:36653
I20240503 14:13:18.164619 9974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.186.8:36653 every 8 connection(s)
I20240503 14:13:18.166208 9847 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 14:13:18.169065 744 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasknKJaW5/build/asan/bin/kudu as pid 9847
I20240503 14:13:18.172868 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:18.192657 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 83c85064eeb84e11a2dc7560b3c80d60 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5]), partition=
I20240503 14:13:18.193080 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c85064eeb84e11a2dc7560b3c80d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.193325 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 83c85064eeb84e11a2dc7560b3c80d60 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5]), partition=
I20240503 14:13:18.193665 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c85064eeb84e11a2dc7560b3c80d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.204253 9975 heartbeater.cc:347] Connected to a master server at 127.0.186.62:45803
I20240503 14:13:18.204573 9975 heartbeater.cc:464] Registering TS with master...
I20240503 14:13:18.205811 9975 heartbeater.cc:511] Master 127.0.186.62:45803 requested a full tablet report, sending...
I20240503 14:13:18.210795 6650 ts_manager.cc:194] Re-registered known tserver with Master: ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:18.212008 6650 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.186.8:37349
I20240503 14:13:18.217917 9991 tablet_bootstrap.cc:492] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:18.219964 9991 tablet_bootstrap.cc:654] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.222447 9991 tablet_bootstrap.cc:492] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:18.222623 9991 ts_tablet_manager.cc:1367] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:18.223574 9991 raft_consensus.cc:348] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.223999 9991 raft_consensus.cc:374] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.224136 9991 raft_consensus.cc:729] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.224457 9991 consensus_queue.cc:260] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.225116 9991 ts_tablet_manager.cc:1398] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:18.228222 9992 tablet_bootstrap.cc:492] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:18.230331 9992 tablet_bootstrap.cc:654] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.238656 9992 tablet_bootstrap.cc:492] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:18.238945 9992 ts_tablet_manager.cc:1367] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240503 14:13:18.239869 9992 raft_consensus.cc:348] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.240209 9992 raft_consensus.cc:374] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.240370 9992 raft_consensus.cc:729] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.240703 9992 consensus_queue.cc:260] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.241333 9992 ts_tablet_manager.cc:1398] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:18.257025 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 83c85064eeb84e11a2dc7560b3c80d60 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5]), partition=
I20240503 14:13:18.258656 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c85064eeb84e11a2dc7560b3c80d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.273224 9995 tablet_bootstrap.cc:492] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:18.276849 9995 tablet_bootstrap.cc:654] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.278555 9995 log.cc:824] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:18.286428 9995 tablet_bootstrap.cc:492] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:18.286751 9995 ts_tablet_manager.cc:1367] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.002s
I20240503 14:13:18.296977 9995 raft_consensus.cc:348] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.297561 9995 raft_consensus.cc:374] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.297696 9995 raft_consensus.cc:729] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.298489 9995 consensus_queue.cc:260] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.300515 9844 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
I20240503 14:13:18.301141 9995 ts_tablet_manager.cc:1398] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
W20240503 14:13:18.382818 9845 tablet.cc:2369] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:18.449680 9872 raft_consensus.cc:482] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:18.449877 9872 raft_consensus.cc:504] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.450939 9872 leader_election.cc:288] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:18.453055 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c85064eeb84e11a2dc7560b3c80d60" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:18.453405 6800 raft_consensus.cc:2454] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:18.454049 6865 leader_election.cc:302] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:18.454429 9872 raft_consensus.cc:2781] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:18.454571 9872 raft_consensus.cc:482] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:18.454689 9872 raft_consensus.cc:3037] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.458405 9872 raft_consensus.cc:504] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.458798 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c85064eeb84e11a2dc7560b3c80d60" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:18.459374 9872 leader_election.cc:288] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:18.459384 9799 raft_consensus.cc:2454] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:18.459992 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c85064eeb84e11a2dc7560b3c80d60" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:18.460244 9798 raft_consensus.cc:3037] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.463328 9798 raft_consensus.cc:2454] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:18.463881 6866 leader_election.cc:302] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:18.464496 9872 raft_consensus.cc:2781] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:18.464639 9872 raft_consensus.cc:686] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:18.464635 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c85064eeb84e11a2dc7560b3c80d60" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:18.464958 9872 consensus_queue.cc:237] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:18.465281 6800 raft_consensus.cc:3037] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:18.468983 6654 catalog_manager.cc:5484] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:18.469362 6800 raft_consensus.cc:2454] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:18.595443 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:18.603166 6800 raft_consensus.cc:1261] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:18.604004 9402 consensus_queue.cc:1035] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:18.604570 9798 raft_consensus.cc:1261] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:18.605464 9402 consensus_queue.cc:1035] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:18.637784 6654 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 83c85064eeb84e11a2dc7560b3c80d60 (table rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5]). Expected version 1 got 0
I20240503 14:13:18.643958 10005 mvcc.cc:205] Tried to move back new op lower bound from 7023597971871596544 to 7023597971316162560. Current Snapshot: MvccSnapshot[applied={T|T < 7023597971871596544}]
I20240503 14:13:18.645507 6650 catalog_manager.cc:5576] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285) does not have the latest schema for tablet 83c85064eeb84e11a2dc7560b3c80d60 (table rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5]). Expected version 1 got 0
I20240503 14:13:18.648702 10006 mvcc.cc:205] Tried to move back new op lower bound from 7023597971871596544 to 7023597971316162560. Current Snapshot: MvccSnapshot[applied={T|T < 7023597971871596544}]
I20240503 14:13:18.652877 10011 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:18.653522 6650 catalog_manager.cc:5576] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285) does not have the latest schema for tablet 83c85064eeb84e11a2dc7560b3c80d60 (table rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5]). Expected version 1 got 0
I20240503 14:13:18.654911 10011 tablet.cc:1620] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: MemRowSet was empty: no flush needed.
I20240503 14:13:18.655180 10013 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:18.658404 10013 tablet.cc:1620] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:18.664067 6654 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 83c85064eeb84e11a2dc7560b3c80d60 (table rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5]). Expected version 1 got 0
I20240503 14:13:18.671852 6659 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:18.673177 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5] alter complete (version 1)
I20240503 14:13:18.674922 6650 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 14:13:18.675217 6650 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 14:13:18.690182 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 01e27516acca49459176362caf73595e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.690268 9779 tablet_service.cc:1467] Processing CreateTablet for tablet b556ba3bf82b40d988c0173f742a0ff2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.690876 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e27516acca49459176362caf73595e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.691216 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b556ba3bf82b40d988c0173f742a0ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.725219 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 565296ab89c24009835aefefa06fd205 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.725932 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565296ab89c24009835aefefa06fd205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.730062 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 7ece38bbe1e5401ba4b10607ffaf9901
I20240503 14:13:18.730798 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet bef030b74c05488a9d8f3ec0df23054a
I20240503 14:13:18.731205 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 176e442aa7f245bc882c15765160ad54
I20240503 14:13:18.731374 9995 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:18.733842 9995 tablet_bootstrap.cc:654] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.736163 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet 176e442aa7f245bc882c15765160ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:18.736040 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ece38bbe1e5401ba4b10607ffaf9901 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:18.736038 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet bef030b74c05488a9d8f3ec0df23054a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:18.737414 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet bef030b74c05488a9d8f3ec0df23054a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:18.737949 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 176e442aa7f245bc882c15765160ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:18.739869 10021 tablet_replica.cc:330] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:18.740363 10021 raft_consensus.cc:2229] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:18.741005 10021 raft_consensus.cc:2258] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.740428 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 01e27516acca49459176362caf73595e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.741274 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ece38bbe1e5401ba4b10607ffaf9901 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:18.741276 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet bef030b74c05488a9d8f3ec0df23054a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:18.742182 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e27516acca49459176362caf73595e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.743829 10021 ts_tablet_manager.cc:1875] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.744202 9910 tablet_service.cc:1467] Processing CreateTablet for tablet b556ba3bf82b40d988c0173f742a0ff2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.744881 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b556ba3bf82b40d988c0173f742a0ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.753171 9995 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:18.753355 9995 ts_tablet_manager.cc:1367] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240503 14:13:18.754468 9995 raft_consensus.cc:348] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:18.754699 9995 raft_consensus.cc:374] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.754792 9995 raft_consensus.cc:729] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.755146 9995 consensus_queue.cc:260] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:18.756459 9995 ts_tablet_manager.cc:1398] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:18.758999 10021 ts_tablet_manager.cc:1888] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.760212 10021 log.cc:1196] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/bef030b74c05488a9d8f3ec0df23054a
W20240503 14:13:18.760185 6867 consensus_peers.cc:485] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:18.761166 10021 ts_tablet_manager.cc:1909] T bef030b74c05488a9d8f3ec0df23054a P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:18.763800 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet bef030b74c05488a9d8f3ec0df23054a (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.766044 10021 tablet_replica.cc:330] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:18.766654 10021 raft_consensus.cc:2229] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:18.769510 10021 raft_consensus.cc:2258] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.770485 10021 ts_tablet_manager.cc:1875] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.771898 10022 tablet_replica.cc:330] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:18.772470 10022 raft_consensus.cc:2229] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:18.772876 10022 raft_consensus.cc:2258] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.774350 10022 ts_tablet_manager.cc:1875] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.777501 10021 ts_tablet_manager.cc:1888] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.781513 10023 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:18.786234 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ece38bbe1e5401ba4b10607ffaf9901 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:18.786490 10023 tablet_bootstrap.cc:654] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.788079 10023 log.cc:824] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:18.791376 7169 tablet_service.cc:1514] Processing DeleteTablet for tablet 176e442aa7f245bc882c15765160ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:18 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:18.797071 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 565296ab89c24009835aefefa06fd205 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.798480 9995 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:18.798877 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565296ab89c24009835aefefa06fd205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.800277 9995 tablet_bootstrap.cc:654] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.804688 10020 tablet_replica.cc:330] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:18.805409 10020 raft_consensus.cc:2229] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:18.806216 10020 raft_consensus.cc:2258] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.809648 10012 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:18.815738 10021 log.cc:1196] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/176e442aa7f245bc882c15765160ad54
I20240503 14:13:18.817958 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 01e27516acca49459176362caf73595e (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.818506 10021 ts_tablet_manager.cc:1909] T 176e442aa7f245bc882c15765160ad54 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:18.819404 6650 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 83c85064eeb84e11a2dc7560b3c80d60 (table rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5]). Expected version 1 got 0
I20240503 14:13:18.819849 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 565296ab89c24009835aefefa06fd205 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.819198 10012 tablet.cc:1620] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:18.820466 9513 tablet_service.cc:1467] Processing CreateTablet for tablet b556ba3bf82b40d988c0173f742a0ff2 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:18.820904 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e27516acca49459176362caf73595e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.821578 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b556ba3bf82b40d988c0173f742a0ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.822378 10020 ts_tablet_manager.cc:1875] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.827893 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 176e442aa7f245bc882c15765160ad54 (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.828336 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565296ab89c24009835aefefa06fd205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:18.829756 10024 tablet_replica.cc:330] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:18.833895 9995 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:18.834187 9995 ts_tablet_manager.cc:1367] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.008s
I20240503 14:13:18.834955 9995 raft_consensus.cc:348] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:18.835304 9995 raft_consensus.cc:374] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.835435 9995 raft_consensus.cc:729] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.835736 9995 consensus_queue.cc:260] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:18.836359 9995 ts_tablet_manager.cc:1398] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:18.830819 10027 tablet_bootstrap.cc:492] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:18.838420 10027 tablet_bootstrap.cc:654] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.846269 10022 ts_tablet_manager.cc:1888] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.846266 10020 ts_tablet_manager.cc:1888] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.846432 10020 log.cc:1196] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/176e442aa7f245bc882c15765160ad54
I20240503 14:13:18.839970 10024 raft_consensus.cc:2229] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:18.851087 10024 raft_consensus.cc:2258] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.852912 10024 ts_tablet_manager.cc:1875] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.855384 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:18.864643 10020 ts_tablet_manager.cc:1909] T 176e442aa7f245bc882c15765160ad54 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:18.865245 10013 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:18.865788 10013 tablet.cc:1620] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:18.867883 10012 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:18.868505 10012 tablet.cc:1620] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:18.864575 10011 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:18.869318 10011 tablet.cc:1620] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: MemRowSet was empty: no flush needed.
I20240503 14:13:18.875362 10022 log.cc:1196] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/7ece38bbe1e5401ba4b10607ffaf9901
I20240503 14:13:18.876104 10022 ts_tablet_manager.cc:1909] T 7ece38bbe1e5401ba4b10607ffaf9901 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:18.876893 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=b16dd66b9fc14d25877a29a568826cc5] alter complete (version 2)
I20240503 14:13:18.879832 10029 tablet_bootstrap.cc:492] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:18.883761 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 7ece38bbe1e5401ba4b10607ffaf9901 (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.884426 10027 tablet_bootstrap.cc:492] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:18.884752 10027 ts_tablet_manager.cc:1367] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20240503 14:13:18.885191 10022 tablet_replica.cc:330] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:18.885604 10022 raft_consensus.cc:2229] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:18.886579 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 176e442aa7f245bc882c15765160ad54 (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.886904 10029 tablet_bootstrap.cc:654] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.891434 10022 raft_consensus.cc:2258] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.892020 10027 raft_consensus.cc:348] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:18.892323 10029 log.cc:824] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:18.892433 10022 ts_tablet_manager.cc:1875] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.893314 10027 raft_consensus.cc:374] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.893631 10027 raft_consensus.cc:729] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.894651 10027 consensus_queue.cc:260] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:18.895992 10027 ts_tablet_manager.cc:1398] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:18.897434 10020 tablet_replica.cc:330] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:18.897722 10020 raft_consensus.cc:2229] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:18.898118 10020 raft_consensus.cc:2258] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.901079 10020 ts_tablet_manager.cc:1875] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.901376 10029 tablet_bootstrap.cc:492] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:18.901744 10029 ts_tablet_manager.cc:1367] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.002s
I20240503 14:13:18.904067 10023 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:18.904429 10023 ts_tablet_manager.cc:1367] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.123s user 0.007s sys 0.035s
I20240503 14:13:18.910620 10034 tablet_bootstrap.cc:492] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:18.912376 10029 raft_consensus.cc:348] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:18.912770 10020 ts_tablet_manager.cc:1888] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.912925 10020 log.cc:1196] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/7ece38bbe1e5401ba4b10607ffaf9901
I20240503 14:13:18.913079 10029 raft_consensus.cc:374] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.913241 10029 raft_consensus.cc:729] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.913362 10020 ts_tablet_manager.cc:1909] T 7ece38bbe1e5401ba4b10607ffaf9901 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:18.914181 10029 consensus_queue.cc:260] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:18.914485 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 7ece38bbe1e5401ba4b10607ffaf9901 (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.920123 10020 tablet_replica.cc:330] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:18.921299 10020 raft_consensus.cc:2229] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:18.922220 10020 raft_consensus.cc:2258] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.923641 10020 ts_tablet_manager.cc:1875] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.925089 10034 tablet_bootstrap.cc:654] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.925541 10022 ts_tablet_manager.cc:1888] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.926088 10029 ts_tablet_manager.cc:1398] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.024s user 0.015s sys 0.004s
I20240503 14:13:18.927491 10022 log.cc:1196] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/bef030b74c05488a9d8f3ec0df23054a
I20240503 14:13:18.928046 10022 ts_tablet_manager.cc:1909] T bef030b74c05488a9d8f3ec0df23054a P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:18.929244 10024 ts_tablet_manager.cc:1888] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.930550 10024 log.cc:1196] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/7ece38bbe1e5401ba4b10607ffaf9901
I20240503 14:13:18.931231 10024 ts_tablet_manager.cc:1909] T 7ece38bbe1e5401ba4b10607ffaf9901 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:18.932265 10029 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:18.932307 10034 log.cc:824] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5: Log is configured to *not* fsync() on all Append() calls
I20240503 14:13:18.933768 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet bef030b74c05488a9d8f3ec0df23054a (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.935747 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 7ece38bbe1e5401ba4b10607ffaf9901 (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.936980 10020 ts_tablet_manager.cc:1888] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.937207 10020 log.cc:1196] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/bef030b74c05488a9d8f3ec0df23054a
I20240503 14:13:18.937763 10020 ts_tablet_manager.cc:1909] T bef030b74c05488a9d8f3ec0df23054a P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:18.938933 10024 tablet_replica.cc:330] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:18.939051 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet bef030b74c05488a9d8f3ec0df23054a (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.939306 10024 raft_consensus.cc:2229] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:18.939632 10024 raft_consensus.cc:2258] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:18.940589 10024 ts_tablet_manager.cc:1875] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:18.941817 10023 raft_consensus.cc:348] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:18.944664 10023 raft_consensus.cc:374] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.944949 10029 tablet_bootstrap.cc:654] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.945001 10023 raft_consensus.cc:729] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.948171 10023 consensus_queue.cc:260] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:18.949627 9975 heartbeater.cc:503] Master 127.0.186.62:45803 was elected leader, sending a full tablet report...
I20240503 14:13:18.951323 10023 ts_tablet_manager.cc:1398] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.047s user 0.012s sys 0.003s
I20240503 14:13:18.954032 10024 ts_tablet_manager.cc:1888] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:18.954128 10034 tablet_bootstrap.cc:492] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:18.954363 10024 log.cc:1196] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/176e442aa7f245bc882c15765160ad54
I20240503 14:13:18.954473 10034 ts_tablet_manager.cc:1367] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.000s
I20240503 14:13:18.955211 10024 ts_tablet_manager.cc:1909] T 176e442aa7f245bc882c15765160ad54 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:18.956647 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 176e442aa7f245bc882c15765160ad54 (table rebalance_test_table_0 [id=ccc823093ced44b998e3c16eca7028cd]) successfully deleted
I20240503 14:13:18.951857 10023 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:18.957540 10029 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:18.957710 10029 ts_tablet_manager.cc:1367] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20240503 14:13:18.958772 10029 raft_consensus.cc:348] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:18.959003 10029 raft_consensus.cc:374] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.959087 10029 raft_consensus.cc:729] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.959175 10023 tablet_bootstrap.cc:654] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.959352 10029 consensus_queue.cc:260] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:18.960754 10029 ts_tablet_manager.cc:1398] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:18.961030 10029 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:18.962846 10029 tablet_bootstrap.cc:654] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:18.964908 10023 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:18.965065 10023 ts_tablet_manager.cc:1367] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240503 14:13:18.965647 10029 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:18.965801 10029 ts_tablet_manager.cc:1367] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 14:13:18.966244 10023 raft_consensus.cc:348] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:18.966449 10023 raft_consensus.cc:374] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.966535 10023 raft_consensus.cc:729] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.966730 10029 raft_consensus.cc:348] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:18.966775 10023 consensus_queue.cc:260] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:18.966982 10029 raft_consensus.cc:374] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.967051 10029 raft_consensus.cc:729] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.967352 10023 ts_tablet_manager.cc:1398] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:18.967269 10029 consensus_queue.cc:260] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:18.967424 10034 raft_consensus.cc:348] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:18.968230 10034 raft_consensus.cc:374] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:18.968358 10034 raft_consensus.cc:729] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:18.968842 10029 ts_tablet_manager.cc:1398] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:18.969237 10034 consensus_queue.cc:260] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:18.974460 10034 ts_tablet_manager.cc:1398] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.020s user 0.015s sys 0.000s
I20240503 14:13:18.975417 9997 raft_consensus.cc:482] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:18.975679 9997 raft_consensus.cc:504] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:18.977756 9997 leader_election.cc:288] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:18.982954 6652 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:19.002012 10011 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:19.002914 10012 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:19.005383 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b556ba3bf82b40d988c0173f742a0ff2" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:19.005956 9930 raft_consensus.cc:2454] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:19.011350 9733 leader_election.cc:302] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:19.011906 9997 raft_consensus.cc:2781] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:19.012042 9997 raft_consensus.cc:482] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:19.012142 9997 raft_consensus.cc:3037] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.012125 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b556ba3bf82b40d988c0173f742a0ff2" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:19.012786 9533 raft_consensus.cc:2454] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:19.015292 6652 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=b16dd66b9fc14d25877a29a568826cc5] alter complete (version 3)
I20240503 14:13:19.015213 9997 raft_consensus.cc:504] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:19.016395 9997 leader_election.cc:288] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:19.016571 10045 raft_consensus.cc:482] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:19.016973 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b556ba3bf82b40d988c0173f742a0ff2" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:19.016860 10045 raft_consensus.cc:504] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:19.017213 9533 raft_consensus.cc:3037] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.020792 9533 raft_consensus.cc:2454] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:19.023532 10045 leader_election.cc:288] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:19.023660 9735 leader_election.cc:302] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:19.024737 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b556ba3bf82b40d988c0173f742a0ff2" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:19.025017 9930 raft_consensus.cc:3037] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.027782 10013 tablet.cc:1722] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:19.035960 9930 raft_consensus.cc:2454] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:19.038008 9997 raft_consensus.cc:2781] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:19.038576 9997 raft_consensus.cc:686] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:19.039119 9997 consensus_queue.cc:237] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:19.051040 6659 catalog_manager.cc:5484] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:19.052500 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565296ab89c24009835aefefa06fd205" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:19.052758 9533 raft_consensus.cc:2454] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:19.053484 9601 leader_election.cc:302] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:19.053925 10045 raft_consensus.cc:2781] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:19.054075 10045 raft_consensus.cc:482] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:19.054191 10045 raft_consensus.cc:3037] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.056620 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565296ab89c24009835aefefa06fd205" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:19.056835 7189 raft_consensus.cc:2454] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:19.057756 10045 raft_consensus.cc:504] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:19.059072 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565296ab89c24009835aefefa06fd205" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:19.059197 10045 leader_election.cc:288] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:19.059321 9533 raft_consensus.cc:3037] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.059758 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565296ab89c24009835aefefa06fd205" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:19.059967 7189 raft_consensus.cc:3037] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.062142 9533 raft_consensus.cc:2454] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:19.062651 9601 leader_election.cc:302] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:19.062749 7189 raft_consensus.cc:2454] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:19.063138 10045 raft_consensus.cc:2781] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:19.064720 10045 raft_consensus.cc:686] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:19.065368 10045 consensus_queue.cc:237] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:19.070375 6659 catalog_manager.cc:5484] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:19.103159 9579 tablet.cc:2369] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 14:13:19.123554 9711 tablet.cc:2369] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:19.124258 6650 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:19.124486 6650 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:19.130311 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 83c85064eeb84e11a2dc7560b3c80d60
I20240503 14:13:19.131444 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c85064eeb84e11a2dc7560b3c80d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:19.131958 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c85064eeb84e11a2dc7560b3c80d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:19.132231 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c85064eeb84e11a2dc7560b3c80d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:19.132560 10022 tablet_replica.cc:330] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:19.132588 10020 tablet_replica.cc:330] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:19.132776 10022 raft_consensus.cc:2229] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:19.132835 10020 raft_consensus.cc:2229] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.132887 10079 tablet_replica.cc:330] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:19.133057 10022 raft_consensus.cc:2258] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.133056 10020 raft_consensus.cc:2258] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.134212 10022 ts_tablet_manager.cc:1875] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.134251 10020 ts_tablet_manager.cc:1875] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.133368 10079 raft_consensus.cc:2229] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.134577 10079 raft_consensus.cc:2258] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.136765 10079 ts_tablet_manager.cc:1875] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.144207 10022 ts_tablet_manager.cc:1888] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:19.144490 10022 log.cc:1196] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/83c85064eeb84e11a2dc7560b3c80d60
I20240503 14:13:19.144585 10079 ts_tablet_manager.cc:1888] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:19.144866 10079 log.cc:1196] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/83c85064eeb84e11a2dc7560b3c80d60
I20240503 14:13:19.145546 10079 ts_tablet_manager.cc:1909] T 83c85064eeb84e11a2dc7560b3c80d60 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:19.147382 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 83c85064eeb84e11a2dc7560b3c80d60 (table rebalancer_dynamic_table_new_name [id=b16dd66b9fc14d25877a29a568826cc5]) successfully deleted
I20240503 14:13:19.151317 10020 ts_tablet_manager.cc:1888] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:19.151497 10020 log.cc:1196] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/83c85064eeb84e11a2dc7560b3c80d60
I20240503 14:13:19.151688 10022 ts_tablet_manager.cc:1909] T 83c85064eeb84e11a2dc7560b3c80d60 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:19.151952 10020 ts_tablet_manager.cc:1909] T 83c85064eeb84e11a2dc7560b3c80d60 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:19.152941 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 83c85064eeb84e11a2dc7560b3c80d60 (table rebalancer_dynamic_table_new_name [id=b16dd66b9fc14d25877a29a568826cc5]) successfully deleted
I20240503 14:13:19.163762 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 83c85064eeb84e11a2dc7560b3c80d60 (table rebalancer_dynamic_table_new_name [id=b16dd66b9fc14d25877a29a568826cc5]) successfully deleted
W20240503 14:13:19.168438 9976 tablet.cc:2369] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:19.233672 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:19.236014 10042 raft_consensus.cc:482] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:19.236239 10042 raft_consensus.cc:504] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:19.238246 10042 leader_election.cc:288] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:19.243530 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 92c55853170b484ca6a21633a3b846a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8]), partition=
I20240503 14:13:19.243677 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 92c55853170b484ca6a21633a3b846a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8]), partition=
I20240503 14:13:19.243677 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 92c55853170b484ca6a21633a3b846a0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8]), partition=
I20240503 14:13:19.243942 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c55853170b484ca6a21633a3b846a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.244056 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c55853170b484ca6a21633a3b846a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.244056 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c55853170b484ca6a21633a3b846a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.249280 6657 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 14:13:19.249523 6657 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 14:13:19.250618 10034 tablet_bootstrap.cc:492] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:19.250988 10023 tablet_bootstrap.cc:492] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:19.251706 9997 raft_consensus.cc:482] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:19.251900 9997 raft_consensus.cc:504] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:19.252476 10034 tablet_bootstrap.cc:654] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.252628 9997 leader_election.cc:288] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:19.253777 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:19.253912 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet de33325421fb42f982d44ca21e00f766
I20240503 14:13:19.254004 9533 raft_consensus.cc:2454] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:19.254396 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:19.254673 9930 raft_consensus.cc:2454] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:19.255014 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet de33325421fb42f982d44ca21e00f766 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:19.255014 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet de33325421fb42f982d44ca21e00f766 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:19.255295 10024 tablet_replica.cc:330] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:19.255504 10024 raft_consensus.cc:2229] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.255781 10024 raft_consensus.cc:2258] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.256505 10024 ts_tablet_manager.cc:1875] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.257858 10020 tablet_replica.cc:330] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:19.258070 10020 raft_consensus.cc:2229] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.258334 10020 raft_consensus.cc:2258] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.259014 10020 ts_tablet_manager.cc:1875] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.260336 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet d6d31c9abf474fff9a7efd9a96679ad4
I20240503 14:13:19.260377 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet de33325421fb42f982d44ca21e00f766 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:19.260797 10022 tablet_replica.cc:330] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:19.260761 9733 leader_election.cc:302] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:19.261113 10022 raft_consensus.cc:2229] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:19.261550 10022 raft_consensus.cc:2258] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.262240 10022 ts_tablet_manager.cc:1875] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.262545 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 58003e797fac46ef8b9689728eebbd43
I20240503 14:13:19.263137 10023 tablet_bootstrap.cc:654] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.263815 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 58003e797fac46ef8b9689728eebbd43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:19.264238 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 58003e797fac46ef8b9689728eebbd43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:19.264402 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d31c9abf474fff9a7efd9a96679ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:19.264408 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 58003e797fac46ef8b9689728eebbd43 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:19.264523 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d31c9abf474fff9a7efd9a96679ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:19.264829 10021 tablet_replica.cc:330] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:19.265014 10021 raft_consensus.cc:2229] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.265290 10021 raft_consensus.cc:2258] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.265997 10021 ts_tablet_manager.cc:1875] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.266382 10034 tablet_bootstrap.cc:492] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:19.266533 10034 ts_tablet_manager.cc:1367] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240503 14:13:19.267557 10034 raft_consensus.cc:348] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.267746 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d31c9abf474fff9a7efd9a96679ad4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:19.267798 10034 raft_consensus.cc:374] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.267884 10034 raft_consensus.cc:729] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.268153 10034 consensus_queue.cc:260] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.268775 10034 ts_tablet_manager.cc:1398] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:19.269176 9997 raft_consensus.cc:2781] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:19.269295 9997 raft_consensus.cc:482] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:19.269413 9997 raft_consensus.cc:3037] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.273381 10022 ts_tablet_manager.cc:1888] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.273587 10022 log.cc:1196] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/de33325421fb42f982d44ca21e00f766
I20240503 14:13:19.274062 10022 ts_tablet_manager.cc:1909] T de33325421fb42f982d44ca21e00f766 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:19.275161 10022 tablet_replica.cc:330] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:19.275319 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet de33325421fb42f982d44ca21e00f766 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.276365 10022 raft_consensus.cc:2229] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.276479 10029 tablet_bootstrap.cc:492] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:19.276639 10022 raft_consensus.cc:2258] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.277330 10022 ts_tablet_manager.cc:1875] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.278096 10029 tablet_bootstrap.cc:654] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:19.281375 9997 raft_consensus.cc:504] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:19.282907 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:19.283174 9533 raft_consensus.cc:3037] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.286159 9533 raft_consensus.cc:2454] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:19.286867 9735 leader_election.cc:302] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:19.287377 10051 raft_consensus.cc:2781] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:19.288465 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:19.289631 10024 ts_tablet_manager.cc:1888] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.290570 9997 leader_election.cc:288] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:19.292023 10024 log.cc:1196] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/de33325421fb42f982d44ca21e00f766
I20240503 14:13:19.292958 10024 ts_tablet_manager.cc:1909] T de33325421fb42f982d44ca21e00f766 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:19.294483 10021 ts_tablet_manager.cc:1888] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.296366 9930 raft_consensus.cc:3037] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.298259 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet de33325421fb42f982d44ca21e00f766 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.299248 10020 ts_tablet_manager.cc:1888] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.299481 9930 raft_consensus.cc:2454] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:19.302693 10024 tablet_replica.cc:330] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:19.294623 10021 log.cc:1196] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/d6d31c9abf474fff9a7efd9a96679ad4
I20240503 14:13:19.303387 10024 raft_consensus.cc:2229] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:19.303457 10021 ts_tablet_manager.cc:1909] T d6d31c9abf474fff9a7efd9a96679ad4 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:19.304095 10020 log.cc:1196] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/de33325421fb42f982d44ca21e00f766
I20240503 14:13:19.305346 10020 ts_tablet_manager.cc:1909] T de33325421fb42f982d44ca21e00f766 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:19.306465 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet d6d31c9abf474fff9a7efd9a96679ad4 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.308712 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet de33325421fb42f982d44ca21e00f766 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.308802 10020 tablet_replica.cc:330] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:19.307444 10024 raft_consensus.cc:2258] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.309041 10020 raft_consensus.cc:2229] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.309249 10020 raft_consensus.cc:2258] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.309999 10020 ts_tablet_manager.cc:1875] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.312325 10029 tablet_bootstrap.cc:492] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:19.312547 10029 ts_tablet_manager.cc:1367] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.005s
I20240503 14:13:19.314074 10029 raft_consensus.cc:348] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.314637 10024 ts_tablet_manager.cc:1875] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.316042 10029 raft_consensus.cc:374] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.316864 10023 tablet_bootstrap.cc:492] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:19.317039 10023 ts_tablet_manager.cc:1367] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.004s
I20240503 14:13:19.317792 10023 raft_consensus.cc:348] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.318019 10023 raft_consensus.cc:374] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:19.318100 10023 raft_consensus.cc:729] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.318454 10020 ts_tablet_manager.cc:1888] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.318378 10023 consensus_queue.cc:260] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.318595 10020 log.cc:1196] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/58003e797fac46ef8b9689728eebbd43
I20240503 14:13:19.319037 10020 ts_tablet_manager.cc:1909] T 58003e797fac46ef8b9689728eebbd43 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:19.319875 10023 ts_tablet_manager.cc:1398] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:19.320129 10029 raft_consensus.cc:729] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:19.320302 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 58003e797fac46ef8b9689728eebbd43 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.320748 10020 tablet_replica.cc:330] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:19.321079 10020 raft_consensus.cc:2229] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:19.321514 10020 raft_consensus.cc:2258] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.322196 10051 raft_consensus.cc:686] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:19.322546 10051 consensus_queue.cc:237] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:19.325168 10020 ts_tablet_manager.cc:1875] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.325671 10029 consensus_queue.cc:260] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.326521 10029 ts_tablet_manager.cc:1398] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 14:13:19.328195 10022 ts_tablet_manager.cc:1888] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.327122 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:19.329020 10022 log.cc:1196] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/58003e797fac46ef8b9689728eebbd43
I20240503 14:13:19.329281 9533 raft_consensus.cc:2379] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea801601713a4d55aee0de9e3473e0bc in current term 1: Already voted for candidate 3efdbd3506274dba846ae40404f0572c in this term.
I20240503 14:13:19.329499 10022 ts_tablet_manager.cc:1909] T 58003e797fac46ef8b9689728eebbd43 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:19.331511 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 58003e797fac46ef8b9689728eebbd43 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.333496 6657 catalog_manager.cc:5484] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:19.335721 10024 ts_tablet_manager.cc:1888] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.335920 10024 log.cc:1196] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/58003e797fac46ef8b9689728eebbd43
I20240503 14:13:19.336880 10024 ts_tablet_manager.cc:1909] T 58003e797fac46ef8b9689728eebbd43 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:19.338037 10024 tablet_replica.cc:330] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:19.338168 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 58003e797fac46ef8b9689728eebbd43 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.338264 10024 raft_consensus.cc:2229] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.338568 10024 raft_consensus.cc:2258] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.339252 10024 ts_tablet_manager.cc:1875] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.339820 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:19.340656 9864 leader_election.cc:302] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:19.341528 10042 raft_consensus.cc:2726] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:19.343287 10020 ts_tablet_manager.cc:1888] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.343410 10020 log.cc:1196] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/d6d31c9abf474fff9a7efd9a96679ad4
I20240503 14:13:19.343914 10020 ts_tablet_manager.cc:1909] T d6d31c9abf474fff9a7efd9a96679ad4 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:19.344990 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet d6d31c9abf474fff9a7efd9a96679ad4 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.345187 10024 ts_tablet_manager.cc:1888] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.345310 10024 log.cc:1196] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/d6d31c9abf474fff9a7efd9a96679ad4
I20240503 14:13:19.345762 10024 ts_tablet_manager.cc:1909] T d6d31c9abf474fff9a7efd9a96679ad4 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:19.346622 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet d6d31c9abf474fff9a7efd9a96679ad4 (table rebalance_test_table_2 [id=152e83c6dd564cd0a8838cbaa4827b00]) successfully deleted
I20240503 14:13:19.383708 7039 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:13:19.395838 9779 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:19.398355 9512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:19.421877 7171 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:13:19.439905 9910 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:19.469272 6911 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:13:19.496345 9645 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240503 14:13:19.505213 6995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.1:34101, user_credentials={real_user=slave}} blocked reactor thread for 79299us
I20240503 14:13:19.539979 10042 raft_consensus.cc:482] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:19.540201 10042 raft_consensus.cc:504] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.541117 10042 leader_election.cc:288] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:19.541895 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c55853170b484ca6a21633a3b846a0" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:19.542155 9533 raft_consensus.cc:2454] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:19.542829 9865 leader_election.cc:302] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:19.543211 10042 raft_consensus.cc:2781] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:19.543330 10042 raft_consensus.cc:482] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:19.543416 10042 raft_consensus.cc:3037] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.546466 10042 raft_consensus.cc:504] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.547982 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c55853170b484ca6a21633a3b846a0" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:19.548226 9533 raft_consensus.cc:3037] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.551380 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c55853170b484ca6a21633a3b846a0" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:19.551801 9665 raft_consensus.cc:2454] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:19.553164 9533 raft_consensus.cc:2454] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:19.553711 9865 leader_election.cc:302] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:19.557545 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c55853170b484ca6a21633a3b846a0" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:19.557786 9665 raft_consensus.cc:3037] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:19.560369 9665 raft_consensus.cc:2454] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:19.561597 10042 leader_election.cc:288] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:19.561846 10042 raft_consensus.cc:2781] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:19.562022 10042 raft_consensus.cc:686] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:19.562479 10042 consensus_queue.cc:237] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:19.568933 6778 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:13:19.578980 10045 consensus_queue.cc:1035] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.579885 10045 consensus_queue.cc:1035] T 565296ab89c24009835aefefa06fd205 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.588407 6657 catalog_manager.cc:5484] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:19.601186 10051 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.617650 9997 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.704491 9997 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.709841 10051 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.754820 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:19.764452 9665 raft_consensus.cc:1261] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:19.765252 9532 raft_consensus.cc:1261] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:19.765897 10042 consensus_queue.cc:1035] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.766201 10111 consensus_queue.cc:1035] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:19.771678 6657 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 14:13:19.772017 6657 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 14:13:19.777941 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 5015820c8edd42dc9e18010e60b4ca19
I20240503 14:13:19.778892 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 837b546bb5e141fb8143da58fbbb084a
I20240503 14:13:19.779098 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 5015820c8edd42dc9e18010e60b4ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:19.779098 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 5015820c8edd42dc9e18010e60b4ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:19.779389 10022 tablet_replica.cc:330] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:19.779533 10021 tablet_replica.cc:330] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:19.779652 10022 raft_consensus.cc:2229] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.779881 10021 raft_consensus.cc:2229] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.779920 10022 raft_consensus.cc:2258] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.780175 10021 raft_consensus.cc:2258] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.780709 10022 ts_tablet_manager.cc:1875] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.781152 10021 ts_tablet_manager.cc:1875] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.783798 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 837b546bb5e141fb8143da58fbbb084a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:19.784137 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 534d2b4eec6547039bc06b9d7837ace4
I20240503 14:13:19.784162 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet 5015820c8edd42dc9e18010e60b4ca19 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:19.784296 10024 tablet_replica.cc:330] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:19.784451 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 837b546bb5e141fb8143da58fbbb084a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:19.784547 10024 raft_consensus.cc:2229] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.784729 10020 tablet_replica.cc:330] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:19.784780 10024 raft_consensus.cc:2258] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.785012 10020 raft_consensus.cc:2229] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:19.785442 10020 raft_consensus.cc:2258] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.785568 10024 ts_tablet_manager.cc:1875] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.786471 10020 ts_tablet_manager.cc:1875] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.788291 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 534d2b4eec6547039bc06b9d7837ace4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:19.788667 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 534d2b4eec6547039bc06b9d7837ace4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:19.789347 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 837b546bb5e141fb8143da58fbbb084a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:19.790406 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 534d2b4eec6547039bc06b9d7837ace4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:19 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:19.795394 10024 ts_tablet_manager.cc:1888] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.796624 10024 log.cc:1196] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/837b546bb5e141fb8143da58fbbb084a
I20240503 14:13:19.797961 10024 ts_tablet_manager.cc:1909] T 837b546bb5e141fb8143da58fbbb084a P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:19.797995 10022 ts_tablet_manager.cc:1888] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.798282 10022 log.cc:1196] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/5015820c8edd42dc9e18010e60b4ca19
I20240503 14:13:19.798734 10022 ts_tablet_manager.cc:1909] T 5015820c8edd42dc9e18010e60b4ca19 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:19.799414 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 837b546bb5e141fb8143da58fbbb084a (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.799860 10024 tablet_replica.cc:330] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:19.799880 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 5015820c8edd42dc9e18010e60b4ca19 (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.800318 10022 tablet_replica.cc:330] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:19.800855 10022 raft_consensus.cc:2229] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.801126 10022 raft_consensus.cc:2258] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.801399 10024 raft_consensus.cc:2229] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.803697 10022 ts_tablet_manager.cc:1875] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.803854 10021 ts_tablet_manager.cc:1888] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.804033 10021 log.cc:1196] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/5015820c8edd42dc9e18010e60b4ca19
W20240503 14:13:19.803994 6734 consensus_peers.cc:485] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:19.804430 10020 ts_tablet_manager.cc:1888] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.805145 10021 ts_tablet_manager.cc:1909] T 5015820c8edd42dc9e18010e60b4ca19 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:19.804611 10020 log.cc:1196] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/5015820c8edd42dc9e18010e60b4ca19
I20240503 14:13:19.805982 10020 ts_tablet_manager.cc:1909] T 5015820c8edd42dc9e18010e60b4ca19 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:19.806780 6655 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 92c55853170b484ca6a21633a3b846a0 (table rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8]). Expected version 1 got 0
I20240503 14:13:19.806964 10020 tablet_replica.cc:330] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:19.807183 10020 raft_consensus.cc:2229] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:19.807760 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 5015820c8edd42dc9e18010e60b4ca19 (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.807803 10024 raft_consensus.cc:2258] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.808641 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 5015820c8edd42dc9e18010e60b4ca19 (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.809548 10122 mvcc.cc:205] Tried to move back new op lower bound from 7023597976624676864 to 7023597975815421952. Current Snapshot: MvccSnapshot[applied={T|T < 7023597976624676864}]
I20240503 14:13:19.812832 10139 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:19.814558 10021 tablet_replica.cc:330] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:19.814518 6655 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 92c55853170b484ca6a21633a3b846a0 (table rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8]). Expected version 1 got 0
I20240503 14:13:19.814785 10021 raft_consensus.cc:2229] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:19.807536 10020 raft_consensus.cc:2258] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.816615 10020 ts_tablet_manager.cc:1875] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.817025 10021 raft_consensus.cc:2258] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.817339 10139 tablet.cc:1620] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:19.818363 10021 ts_tablet_manager.cc:1875] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.820883 10024 ts_tablet_manager.cc:1875] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.822373 10022 ts_tablet_manager.cc:1888] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.826936 10022 log.cc:1196] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/837b546bb5e141fb8143da58fbbb084a
I20240503 14:13:19.827625 10022 ts_tablet_manager.cc:1909] T 837b546bb5e141fb8143da58fbbb084a P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:19.828601 6657 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 92c55853170b484ca6a21633a3b846a0 (table rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8]). Expected version 1 got 0
I20240503 14:13:19.828740 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 837b546bb5e141fb8143da58fbbb084a (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.833068 10021 ts_tablet_manager.cc:1888] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.836153 10020 ts_tablet_manager.cc:1888] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.836220 6657 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8] alter complete (version 1)
I20240503 14:13:19.836988 10020 log.cc:1196] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/837b546bb5e141fb8143da58fbbb084a
I20240503 14:13:19.837262 10021 log.cc:1196] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/534d2b4eec6547039bc06b9d7837ace4
I20240503 14:13:19.837591 10141 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:19.838451 10021 ts_tablet_manager.cc:1909] T 534d2b4eec6547039bc06b9d7837ace4 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:19.838497 6657 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 92c55853170b484ca6a21633a3b846a0 (table rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8]). Expected version 1 got 0
I20240503 14:13:19.838994 10020 ts_tablet_manager.cc:1909] T 837b546bb5e141fb8143da58fbbb084a P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:19.840165 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 837b546bb5e141fb8143da58fbbb084a (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.841240 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 534d2b4eec6547039bc06b9d7837ace4 (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.841854 10141 tablet.cc:1620] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:19.842116 10020 tablet_replica.cc:330] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:19.842780 10020 raft_consensus.cc:2229] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:19.843029 10020 raft_consensus.cc:2258] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:19.845499 6657 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 92c55853170b484ca6a21633a3b846a0 (table rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8]). Expected version 1 got 0
I20240503 14:13:19.843945 10138 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:19.845718 10020 ts_tablet_manager.cc:1875] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:19.849507 10138 tablet.cc:1620] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:19.852779 10020 ts_tablet_manager.cc:1888] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.852995 10020 log.cc:1196] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/534d2b4eec6547039bc06b9d7837ace4
I20240503 14:13:19.853467 10020 ts_tablet_manager.cc:1909] T 534d2b4eec6547039bc06b9d7837ace4 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:19.854588 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 534d2b4eec6547039bc06b9d7837ace4 (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.856062 10024 ts_tablet_manager.cc:1888] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:19.858289 10024 log.cc:1196] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/534d2b4eec6547039bc06b9d7837ace4
I20240503 14:13:19.858722 10024 ts_tablet_manager.cc:1909] T 534d2b4eec6547039bc06b9d7837ace4 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:19.860002 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 534d2b4eec6547039bc06b9d7837ace4 (table rebalance_test_table_4 [id=1107326d66fb41a289767226ccdacf7d]) successfully deleted
I20240503 14:13:19.939973 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:19.948674 10138 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:19.949220 10138 tablet.cc:1620] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:19.949975 10141 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:19.950485 10141 tablet.cc:1620] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:19.952086 10139 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:19.952510 10139 tablet.cc:1620] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:19.967931 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=dd6f3e0fcc30488482cc8d12592a5cc8] alter complete (version 2)
I20240503 14:13:19.978816 6657 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:19.996137 9779 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6c0c0b6c249208e67400face0b9a6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:19.996677 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6c0c0b6c249208e67400face0b9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:19.999251 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 32d91b6f4bb244c78bc6fef8b837f964 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:19.999748 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d91b6f4bb244c78bc6fef8b837f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.000049 7039 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6c0c0b6c249208e67400face0b9a6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.000505 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6c0c0b6c249208e67400face0b9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.003031 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 0b18e1d58117421c82ba86c08cd1253b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.003420 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b18e1d58117421c82ba86c08cd1253b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.004573 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 0b18e1d58117421c82ba86c08cd1253b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.005018 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b18e1d58117421c82ba86c08cd1253b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.010715 9910 tablet_service.cc:1467] Processing CreateTablet for tablet d4d6c0c0b6c249208e67400face0b9a6 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.011250 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6c0c0b6c249208e67400face0b9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.014370 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 32d91b6f4bb244c78bc6fef8b837f964 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.014916 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d91b6f4bb244c78bc6fef8b837f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.018702 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 32d91b6f4bb244c78bc6fef8b837f964 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.019328 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d91b6f4bb244c78bc6fef8b837f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.020156 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 0b18e1d58117421c82ba86c08cd1253b (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:20.020712 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b18e1d58117421c82ba86c08cd1253b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.051730 10172 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:20.054266 10170 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:20.056855 10170 tablet_bootstrap.cc:654] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.057523 10171 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:20.059618 10172 tablet_bootstrap.cc:654] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.060622 10169 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:20.061316 10171 tablet_bootstrap.cc:654] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.062480 10168 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:20.066546 10172 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:20.066797 10172 ts_tablet_manager.cc:1367] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20240503 14:13:20.067931 10171 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:20.068045 10168 tablet_bootstrap.cc:654] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.068260 10171 ts_tablet_manager.cc:1367] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240503 14:13:20.069295 10171 raft_consensus.cc:348] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.070814 10171 raft_consensus.cc:374] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.071269 6655 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:20.071628 10172 raft_consensus.cc:348] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.071950 10172 raft_consensus.cc:374] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.072099 10172 raft_consensus.cc:729] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.072427 10172 consensus_queue.cc:260] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.074914 10171 raft_consensus.cc:729] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.076076 10169 tablet_bootstrap.cc:654] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.076089 10170 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:20.076249 10170 ts_tablet_manager.cc:1367] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.006s
I20240503 14:13:20.077066 10170 raft_consensus.cc:348] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.077281 10170 raft_consensus.cc:374] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.078189 10172 ts_tablet_manager.cc:1398] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:20.078228 10170 raft_consensus.cc:729] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.078644 10172 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:20.078518 10170 consensus_queue.cc:260] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.080564 10172 tablet_bootstrap.cc:654] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.081979 10170 ts_tablet_manager.cc:1398] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:20.085271 10170 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:20.085636 10171 consensus_queue.cc:260] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.087862 10170 tablet_bootstrap.cc:654] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.090395 10138 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:20.097000 10139 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:20.103039 10171 ts_tablet_manager.cc:1398] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.034s user 0.000s sys 0.002s
I20240503 14:13:20.103468 10171 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:20.104897 10172 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:20.099751 10141 tablet.cc:1722] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:20.105103 10172 ts_tablet_manager.cc:1367] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.012s
I20240503 14:13:20.106806 10171 tablet_bootstrap.cc:654] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.107225 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=dd6f3e0fcc30488482cc8d12592a5cc8] alter complete (version 3)
I20240503 14:13:20.107290 10172 raft_consensus.cc:348] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.107544 10172 raft_consensus.cc:374] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.107648 10172 raft_consensus.cc:729] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.107949 10172 consensus_queue.cc:260] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.108799 10172 ts_tablet_manager.cc:1398] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:20.125610 10168 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:20.125804 10168 ts_tablet_manager.cc:1367] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.000s
I20240503 14:13:20.126538 10168 raft_consensus.cc:348] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.126700 10168 raft_consensus.cc:374] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.126782 10168 raft_consensus.cc:729] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.127041 10168 consensus_queue.cc:260] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.127527 10168 ts_tablet_manager.cc:1398] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:20.128110 10169 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:20.131429 10169 ts_tablet_manager.cc:1367] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.000s
I20240503 14:13:20.132457 10169 raft_consensus.cc:348] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.132763 10169 raft_consensus.cc:374] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.132887 10169 raft_consensus.cc:729] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.133168 10169 consensus_queue.cc:260] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.133760 10169 ts_tablet_manager.cc:1398] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:20.136662 10168 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:20.138546 10168 tablet_bootstrap.cc:654] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.140406 10170 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:20.140686 10170 ts_tablet_manager.cc:1367] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.006s
I20240503 14:13:20.141553 10170 raft_consensus.cc:348] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.141881 10170 raft_consensus.cc:374] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.142006 10170 raft_consensus.cc:729] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.142436 10170 consensus_queue.cc:260] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.143110 10170 ts_tablet_manager.cc:1398] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:20.144080 10171 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:20.144316 10171 ts_tablet_manager.cc:1367] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.005s
I20240503 14:13:20.145184 10171 raft_consensus.cc:348] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.145498 10171 raft_consensus.cc:374] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.145610 10171 raft_consensus.cc:729] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.145879 10171 consensus_queue.cc:260] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.146526 10171 ts_tablet_manager.cc:1398] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:20.160750 10168 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:20.161052 10168 ts_tablet_manager.cc:1367] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.009s
I20240503 14:13:20.161698 10168 raft_consensus.cc:348] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.161902 10168 raft_consensus.cc:374] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.161981 10168 raft_consensus.cc:729] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.162249 10168 consensus_queue.cc:260] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.162933 10168 ts_tablet_manager.cc:1398] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:20.187492 10111 raft_consensus.cc:482] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.187749 10111 raft_consensus.cc:504] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.188637 10111 leader_election.cc:288] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:20.189512 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:20.189754 9665 raft_consensus.cc:2454] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:20.190419 9863 leader_election.cc:302] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:20.190827 10111 raft_consensus.cc:2781] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.190934 10111 raft_consensus.cc:482] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.191049 10111 raft_consensus.cc:3037] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.191072 10037 raft_consensus.cc:482] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.191306 10037 raft_consensus.cc:504] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.193399 10111 raft_consensus.cc:504] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.193745 10037 leader_election.cc:288] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:20.194941 10111 leader_election.cc:288] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:20.195037 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:20.195232 9665 raft_consensus.cc:3037] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:20.195338 7107 tablet.cc:2369] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:20.198094 9665 raft_consensus.cc:2454] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:20.198707 9863 leader_election.cc:302] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:20.199661 10111 raft_consensus.cc:2781] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:20.200245 10111 raft_consensus.cc:686] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:20.200572 10111 consensus_queue.cc:237] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:20.203294 9779 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:20.218052 6651 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:20.218348 6651 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:20.222239 6651 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 92c55853170b484ca6a21633a3b846a0
I20240503 14:13:20.222591 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:20.222978 9532 raft_consensus.cc:2454] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:20.223853 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c55853170b484ca6a21633a3b846a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:20.223909 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c55853170b484ca6a21633a3b846a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:20.224231 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:20.224490 9532 raft_consensus.cc:3037] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.230604 9532 raft_consensus.cc:2454] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:20.253419 9715 raft_consensus.cc:482] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.253685 9715 raft_consensus.cc:504] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.254889 10203 tablet_replica.cc:330] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:20.254956 9715 leader_election.cc:288] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:20.255648 10204 tablet_replica.cc:330] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:20.256237 10203 raft_consensus.cc:2229] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.258072 10203 raft_consensus.cc:2258] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.264832 7039 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:13:20.265090 10204 raft_consensus.cc:2229] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.265597 10204 raft_consensus.cc:2258] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.270694 10203 ts_tablet_manager.cc:1875] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.270972 10204 ts_tablet_manager.cc:1875] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:13:20.271222 7125 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.2:45285, user_credentials={real_user=slave}} blocked reactor thread for 66900.8us
I20240503 14:13:20.274740 6651 catalog_manager.cc:5484] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:20.288271 6651 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 14:13:20.288647 6651 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 14:13:20.290693 9512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:20.294024 6651 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 2448fc37870d4d3b9bf267da36780f94
I20240503 14:13:20.294870 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 2448fc37870d4d3b9bf267da36780f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:20.295182 10020 tablet_replica.cc:330] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:20.295377 10020 raft_consensus.cc:2229] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:20.295781 10020 raft_consensus.cc:2258] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.296535 10020 ts_tablet_manager.cc:1875] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.301119 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2448fc37870d4d3b9bf267da36780f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:20.301513 10024 tablet_replica.cc:330] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:20.301764 10024 raft_consensus.cc:2229] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.302057 10024 raft_consensus.cc:2258] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.302959 10024 ts_tablet_manager.cc:1875] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.302552 6651 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 09400f720f464ef8b8bec4549826632e
I20240503 14:13:20.304488 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 09400f720f464ef8b8bec4549826632e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:20.307756 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 2448fc37870d4d3b9bf267da36780f94 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:20.308044 10022 tablet_replica.cc:330] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:20.308290 10022 raft_consensus.cc:2229] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.308537 10022 raft_consensus.cc:2258] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.308624 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 09400f720f464ef8b8bec4549826632e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:20.309127 6651 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 310a9ef153524468abb5a00808d186c6
I20240503 14:13:20.309343 10022 ts_tablet_manager.cc:1875] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.310065 10024 ts_tablet_manager.cc:1888] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.310019 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 310a9ef153524468abb5a00808d186c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:20.310195 10024 log.cc:1196] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/2448fc37870d4d3b9bf267da36780f94
I20240503 14:13:20.310576 10024 ts_tablet_manager.cc:1909] T 2448fc37870d4d3b9bf267da36780f94 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:20.311743 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 2448fc37870d4d3b9bf267da36780f94 (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.312466 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 310a9ef153524468abb5a00808d186c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:20.312732 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 310a9ef153524468abb5a00808d186c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:20.312991 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 09400f720f464ef8b8bec4549826632e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:20.313298 10021 tablet_replica.cc:330] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:20.313541 10021 raft_consensus.cc:2229] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.313746 10021 raft_consensus.cc:2258] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.314463 10021 ts_tablet_manager.cc:1875] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.318868 10020 ts_tablet_manager.cc:1888] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.319123 10020 log.cc:1196] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/2448fc37870d4d3b9bf267da36780f94
I20240503 14:13:20.321363 10024 tablet_replica.cc:330] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:20.321600 10024 raft_consensus.cc:2229] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:20.322039 10024 raft_consensus.cc:2258] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.323094 10024 ts_tablet_manager.cc:1875] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.324880 6911 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:13:20.326687 10022 ts_tablet_manager.cc:1888] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.326882 10022 log.cc:1196] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/2448fc37870d4d3b9bf267da36780f94
I20240503 14:13:20.327160 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c55853170b484ca6a21633a3b846a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:20.327335 10022 ts_tablet_manager.cc:1909] T 2448fc37870d4d3b9bf267da36780f94 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:20.328884 10037 raft_consensus.cc:482] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.329056 10037 raft_consensus.cc:504] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.329838 10037 leader_election.cc:288] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:20.330183 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:20.343436 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 7840eb058d94456083c4b6ff99e9b38c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794]), partition=
I20240503 14:13:20.343987 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7840eb058d94456083c4b6ff99e9b38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.344832 10024 ts_tablet_manager.cc:1888] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.344960 10024 log.cc:1196] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/09400f720f464ef8b8bec4549826632e
I20240503 14:13:20.345022 10021 ts_tablet_manager.cc:1888] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.345165 10021 log.cc:1196] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/09400f720f464ef8b8bec4549826632e
I20240503 14:13:20.345472 10024 ts_tablet_manager.cc:1909] T 09400f720f464ef8b8bec4549826632e P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:20.345577 10021 ts_tablet_manager.cc:1909] T 09400f720f464ef8b8bec4549826632e P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:20.346557 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 09400f720f464ef8b8bec4549826632e (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.346557 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 09400f720f464ef8b8bec4549826632e (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.349655 10204 ts_tablet_manager.cc:1888] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:20.352092 10022 tablet_replica.cc:330] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:20.352602 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 2448fc37870d4d3b9bf267da36780f94 (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.355389 10022 raft_consensus.cc:2229] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:20.355854 10022 raft_consensus.cc:2258] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.356611 10203 ts_tablet_manager.cc:1888] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:20.357496 10022 ts_tablet_manager.cc:1875] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.364331 10203 log.cc:1196] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/92c55853170b484ca6a21633a3b846a0
I20240503 14:13:20.365826 10203 ts_tablet_manager.cc:1909] T 92c55853170b484ca6a21633a3b846a0 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:20.366390 10204 log.cc:1196] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/92c55853170b484ca6a21633a3b846a0
I20240503 14:13:20.367885 10024 tablet_replica.cc:330] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:20.368150 10024 raft_consensus.cc:2229] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.368400 10024 raft_consensus.cc:2258] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.368417 10204 ts_tablet_manager.cc:1909] T 92c55853170b484ca6a21633a3b846a0 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:20.369171 10024 ts_tablet_manager.cc:1875] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.369211 10172 tablet_bootstrap.cc:492] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:20.371277 10172 tablet_bootstrap.cc:654] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.372848 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 92c55853170b484ca6a21633a3b846a0 (table rebalancer_dynamic_table_new_name [id=dd6f3e0fcc30488482cc8d12592a5cc8]) successfully deleted
I20240503 14:13:20.373983 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 7840eb058d94456083c4b6ff99e9b38c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794]), partition=
I20240503 14:13:20.374455 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7840eb058d94456083c4b6ff99e9b38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.380488 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:20.381969 10020 ts_tablet_manager.cc:1909] T 2448fc37870d4d3b9bf267da36780f94 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:20.382836 10171 tablet_bootstrap.cc:492] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:20.383258 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 2448fc37870d4d3b9bf267da36780f94 (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.383679 10020 tablet_replica.cc:330] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:20.383901 10020 raft_consensus.cc:2229] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.384124 10020 raft_consensus.cc:2258] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.384750 10171 tablet_bootstrap.cc:654] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.385655 10208 tablet_replica.cc:330] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:20.386209 10208 raft_consensus.cc:2229] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:20.387007 10208 raft_consensus.cc:2258] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.388837 10051 raft_consensus.cc:482] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.389029 10051 raft_consensus.cc:504] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.389899 10051 leader_election.cc:288] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:20.391499 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b18e1d58117421c82ba86c08cd1253b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:20.391868 9532 raft_consensus.cc:2454] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:20.392450 10172 tablet_bootstrap.cc:492] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:20.392514 9735 leader_election.cc:302] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:20.392627 10172 ts_tablet_manager.cc:1367] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.008s
I20240503 14:13:20.393015 10051 raft_consensus.cc:2781] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.393113 10051 raft_consensus.cc:482] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.393218 10051 raft_consensus.cc:3037] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.393946 10172 raft_consensus.cc:348] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.394127 10172 raft_consensus.cc:374] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.394207 10172 raft_consensus.cc:729] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.394444 10172 consensus_queue.cc:260] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.396190 10051 raft_consensus.cc:504] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.396468 10172 ts_tablet_manager.cc:1398] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:20.397439 10051 leader_election.cc:288] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:20.397783 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b18e1d58117421c82ba86c08cd1253b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:20.398029 9532 raft_consensus.cc:3037] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.400964 9532 raft_consensus.cc:2454] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:20.401446 9735 leader_election.cc:302] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:20.401849 10051 raft_consensus.cc:2781] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:20.402410 10051 raft_consensus.cc:686] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:20.402755 10051 consensus_queue.cc:237] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.411834 10022 ts_tablet_manager.cc:1888] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.412879 10022 log.cc:1196] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/310a9ef153524468abb5a00808d186c6
I20240503 14:13:20.413568 10022 ts_tablet_manager.cc:1909] T 310a9ef153524468abb5a00808d186c6 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:20.415846 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 310a9ef153524468abb5a00808d186c6 (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.424603 10024 ts_tablet_manager.cc:1888] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.424768 10024 log.cc:1196] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/310a9ef153524468abb5a00808d186c6
I20240503 14:13:20.430495 10024 ts_tablet_manager.cc:1909] T 310a9ef153524468abb5a00808d186c6 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:20.431892 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 92c55853170b484ca6a21633a3b846a0 (table rebalancer_dynamic_table_new_name [id=dd6f3e0fcc30488482cc8d12592a5cc8]) successfully deleted
I20240503 14:13:20.432816 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 7840eb058d94456083c4b6ff99e9b38c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794]), partition=
I20240503 14:13:20.433238 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7840eb058d94456083c4b6ff99e9b38c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:20.440151 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:20.440573 9665 raft_consensus.cc:2379] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b7e36bf354f480c90859cb8867b59cf in current term 1: Already voted for candidate ea801601713a4d55aee0de9e3473e0bc in this term.
I20240503 14:13:20.440694 10170 tablet_bootstrap.cc:492] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:20.440173 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b18e1d58117421c82ba86c08cd1253b" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:20.441176 9664 raft_consensus.cc:2454] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:20.442754 10170 tablet_bootstrap.cc:654] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:20.448249 9467 leader_election.cc:302] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:20.449126 10037 raft_consensus.cc:2726] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:20.449896 9467 leader_election.cc:302] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:20.450263 10037 raft_consensus.cc:2741] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20240503 14:13:20.452257 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 310a9ef153524468abb5a00808d186c6 (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.453068 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b18e1d58117421c82ba86c08cd1253b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:20.453302 9664 raft_consensus.cc:2454] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:20.453573 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b18e1d58117421c82ba86c08cd1253b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:20.453752 9665 raft_consensus.cc:3037] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.456604 9665 raft_consensus.cc:2454] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:20.456946 10170 tablet_bootstrap.cc:492] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:20.457113 10170 ts_tablet_manager.cc:1367] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.010s
I20240503 14:13:20.460850 10020 ts_tablet_manager.cc:1875] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.460831 10170 raft_consensus.cc:348] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.461063 10170 raft_consensus.cc:374] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.461156 10170 raft_consensus.cc:729] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.461544 10170 consensus_queue.cc:260] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.463084 10170 ts_tablet_manager.cc:1398] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:13:20.411087 10171 tablet_bootstrap.cc:492] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:20.464272 10171 ts_tablet_manager.cc:1367] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.082s user 0.007s sys 0.003s
I20240503 14:13:20.465143 10171 raft_consensus.cc:348] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.465341 10171 raft_consensus.cc:374] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:20.465427 10171 raft_consensus.cc:729] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:20.465495 10208 ts_tablet_manager.cc:1875] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.465651 10171 consensus_queue.cc:260] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.466156 10171 ts_tablet_manager.cc:1398] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:20.423072 10111 raft_consensus.cc:482] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.466426 10111 raft_consensus.cc:504] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.467234 10111 leader_election.cc:288] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:20.462958 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c0c0b6c249208e67400face0b9a6" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:20.468034 9930 raft_consensus.cc:2454] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:20.467988 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c0c0b6c249208e67400face0b9a6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:20.468278 9799 raft_consensus.cc:2454] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:20.468741 9864 leader_election.cc:302] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:20.469039 10042 raft_consensus.cc:2781] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.469130 10042 raft_consensus.cc:482] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.469228 10042 raft_consensus.cc:3037] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.471122 6651 catalog_manager.cc:5484] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:20.474012 10042 raft_consensus.cc:504] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.475505 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c0c0b6c249208e67400face0b9a6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:20.475786 9799 raft_consensus.cc:3037] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.479317 9799 raft_consensus.cc:2454] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:20.480043 9864 leader_election.cc:302] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:20.480489 10111 raft_consensus.cc:2781] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:20.481292 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b18e1d58117421c82ba86c08cd1253b" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:20.482915 10042 leader_election.cc:288] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:20.483573 10208 ts_tablet_manager.cc:1888] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:20.483896 10208 log.cc:1196] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/92c55853170b484ca6a21633a3b846a0
I20240503 14:13:20.484006 6995 leader_election.cc:302] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:20.484745 10208 ts_tablet_manager.cc:1909] T 92c55853170b484ca6a21633a3b846a0 P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:20.485546 10109 raft_consensus.cc:2781] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.485669 10109 raft_consensus.cc:482] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.485783 10109 raft_consensus.cc:3037] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.487030 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 92c55853170b484ca6a21633a3b846a0 (table rebalancer_dynamic_table_new_name [id=dd6f3e0fcc30488482cc8d12592a5cc8]) successfully deleted
I20240503 14:13:20.488448 10109 raft_consensus.cc:504] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.489180 10109 leader_election.cc:288] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:20.489363 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c0c0b6c249208e67400face0b9a6" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:20.489789 9799 raft_consensus.cc:2379] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09723f236d8044538d920a37b5d9aa8c in current term 1: Already voted for candidate ea801601713a4d55aee0de9e3473e0bc in this term.
I20240503 14:13:20.490240 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c0c0b6c249208e67400face0b9a6" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:20.491027 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c0c0b6c249208e67400face0b9a6" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:20.491258 9799 raft_consensus.cc:2379] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 09723f236d8044538d920a37b5d9aa8c in current term 1: Already voted for candidate ea801601713a4d55aee0de9e3473e0bc in this term.
I20240503 14:13:20.498199 7171 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:20.514501 10111 raft_consensus.cc:686] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:20.514995 10111 consensus_queue.cc:237] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:20.522080 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c0c0b6c249208e67400face0b9a6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:20.522457 7060 raft_consensus.cc:2379] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea801601713a4d55aee0de9e3473e0bc in current term 1: Already voted for candidate 09723f236d8044538d920a37b5d9aa8c in this term.
I20240503 14:13:20.522636 6995 leader_election.cc:302] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c; no voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:20.522810 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d6c0c0b6c249208e67400face0b9a6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:20.523069 7058 raft_consensus.cc:2379] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ea801601713a4d55aee0de9e3473e0bc in current term 1: Already voted for candidate 09723f236d8044538d920a37b5d9aa8c in this term.
I20240503 14:13:20.523175 9715 raft_consensus.cc:2726] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 14:13:20.524458 6651 catalog_manager.cc:5484] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:20.526890 10020 ts_tablet_manager.cc:1888] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.527070 10020 log.cc:1196] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/09400f720f464ef8b8bec4549826632e
I20240503 14:13:20.527657 10020 ts_tablet_manager.cc:1909] T 09400f720f464ef8b8bec4549826632e P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:20.528699 10020 tablet_replica.cc:330] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:20.528770 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 09400f720f464ef8b8bec4549826632e (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.528925 10020 raft_consensus.cc:2229] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.529165 10020 raft_consensus.cc:2258] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.529901 10020 ts_tablet_manager.cc:1875] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.538566 10020 ts_tablet_manager.cc:1888] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.538726 10020 log.cc:1196] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/310a9ef153524468abb5a00808d186c6
I20240503 14:13:20.539240 10020 ts_tablet_manager.cc:1909] T 310a9ef153524468abb5a00808d186c6 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:20.540297 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 310a9ef153524468abb5a00808d186c6 (table rebalance_test_table_6 [id=d173c544325443e4acb5c3180a96928b]) successfully deleted
I20240503 14:13:20.550189 9909 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:20.572144 9644 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:20.615746 6779 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:13:20.618424 10113 raft_consensus.cc:482] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.618609 10113 raft_consensus.cc:504] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.619520 10113 leader_election.cc:288] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:20.620285 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7840eb058d94456083c4b6ff99e9b38c" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:20.620597 9532 raft_consensus.cc:2454] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:20.621443 9601 leader_election.cc:302] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:20.621934 10113 raft_consensus.cc:2781] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:20.622084 10113 raft_consensus.cc:482] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:20.622190 10113 raft_consensus.cc:3037] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.626365 10113 raft_consensus.cc:504] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.628567 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7840eb058d94456083c4b6ff99e9b38c" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:20.628861 9532 raft_consensus.cc:3037] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.633121 9532 raft_consensus.cc:2454] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:20.633644 9601 leader_election.cc:302] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:20.635147 10111 raft_consensus.cc:482] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:20.635350 10111 raft_consensus.cc:504] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.636263 10111 leader_election.cc:288] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:20.637185 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7840eb058d94456083c4b6ff99e9b38c" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:20.637645 9532 raft_consensus.cc:2379] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea801601713a4d55aee0de9e3473e0bc in current term 1: Already voted for candidate f03dc37fe3234033bc665bed4f1b51f5 in this term.
I20240503 14:13:20.638933 10113 leader_election.cc:288] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:20.643918 10113 raft_consensus.cc:2781] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:20.644088 10113 raft_consensus.cc:686] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:20.644395 10113 consensus_queue.cc:237] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:20.648057 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7840eb058d94456083c4b6ff99e9b38c" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:20.648262 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7840eb058d94456083c4b6ff99e9b38c" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:20.648347 9930 raft_consensus.cc:2454] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:20.655164 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7840eb058d94456083c4b6ff99e9b38c" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:20.656059 9863 leader_election.cc:302] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:20.656489 10111 raft_consensus.cc:3037] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:20.659256 10111 raft_consensus.cc:2726] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:20.662035 6651 catalog_manager.cc:5484] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:20.716761 10111 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.736456 10111 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.769901 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:20.776866 9929 raft_consensus.cc:1261] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:20.778092 10226 consensus_queue.cc:1035] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.778187 9532 raft_consensus.cc:1261] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:20.778712 10226 consensus_queue.cc:1035] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.786590 10236 mvcc.cc:205] Tried to move back new op lower bound from 7023597980775026688 to 7023597980248072192. Current Snapshot: MvccSnapshot[applied={T|T < 7023597980775026688}]
I20240503 14:13:20.791903 6651 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 7840eb058d94456083c4b6ff99e9b38c (table rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794]). Expected version 1 got 0
I20240503 14:13:20.795722 6650 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 7840eb058d94456083c4b6ff99e9b38c (table rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794]). Expected version 1 got 0
I20240503 14:13:20.795967 10247 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:20.796932 10247 tablet.cc:1620] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:20.798756 6650 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 7840eb058d94456083c4b6ff99e9b38c (table rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794]). Expected version 1 got 0
I20240503 14:13:20.799170 10248 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:20.799819 10248 tablet.cc:1620] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:20.800984 6650 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 7840eb058d94456083c4b6ff99e9b38c (table rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794]). Expected version 1 got 0
I20240503 14:13:20.815567 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794] alter complete (version 1)
I20240503 14:13:20.816677 6650 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 14:13:20.817049 6650 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 14:13:20.820770 10251 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:20.821625 10251 tablet.cc:1620] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:20.822129 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 46d94c6f01544219a5602b3f06cb35b7
I20240503 14:13:20.822850 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d94c6f01544219a5602b3f06cb35b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:20.823153 10024 tablet_replica.cc:330] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:20.823379 10024 raft_consensus.cc:2229] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.823431 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 8d111a5f08914420a7a429987c9699e9
I20240503 14:13:20.823714 10024 raft_consensus.cc:2258] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.824131 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 6f8f4639ce314b34b88f67315f3e4fa7
I20240503 14:13:20.824129 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d111a5f08914420a7a429987c9699e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:20.824515 10024 ts_tablet_manager.cc:1875] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.825268 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8f4639ce314b34b88f67315f3e4fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:20.825896 6651 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 7840eb058d94456083c4b6ff99e9b38c (table rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794]). Expected version 1 got 0
I20240503 14:13:20.827337 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d94c6f01544219a5602b3f06cb35b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:20.827494 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d111a5f08914420a7a429987c9699e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:20.827512 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 46d94c6f01544219a5602b3f06cb35b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:20.828135 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8f4639ce314b34b88f67315f3e4fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:20.828212 10020 tablet_replica.cc:330] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:20.828444 10020 raft_consensus.cc:2229] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.828680 10020 raft_consensus.cc:2258] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.829335 10020 ts_tablet_manager.cc:1875] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.831784 10021 tablet_replica.cc:330] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:20.831992 10021 raft_consensus.cc:2229] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:20.832343 10021 raft_consensus.cc:2258] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.833083 10021 ts_tablet_manager.cc:1875] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.835729 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d111a5f08914420a7a429987c9699e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:20.835994 10022 tablet_replica.cc:330] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:20.836208 10022 raft_consensus.cc:2229] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.836448 10022 raft_consensus.cc:2258] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.836550 10020 ts_tablet_manager.cc:1888] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.836678 10020 log.cc:1196] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/8d111a5f08914420a7a429987c9699e9
I20240503 14:13:20.837108 10020 ts_tablet_manager.cc:1909] T 8d111a5f08914420a7a429987c9699e9 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:20.837275 10022 ts_tablet_manager.cc:1875] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.838085 10020 tablet_replica.cc:330] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:20.838245 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 8d111a5f08914420a7a429987c9699e9 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.838279 10020 raft_consensus.cc:2229] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.838496 10020 raft_consensus.cc:2258] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.839113 10020 ts_tablet_manager.cc:1875] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.840499 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8f4639ce314b34b88f67315f3e4fa7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:20 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:20.843580 10024 ts_tablet_manager.cc:1888] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.844238 10022 ts_tablet_manager.cc:1888] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.845197 10022 log.cc:1196] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/46d94c6f01544219a5602b3f06cb35b7
I20240503 14:13:20.845733 10022 ts_tablet_manager.cc:1909] T 46d94c6f01544219a5602b3f06cb35b7 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:20.846973 10021 ts_tablet_manager.cc:1888] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.847090 10021 log.cc:1196] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/6f8f4639ce314b34b88f67315f3e4fa7
I20240503 14:13:20.847218 10022 tablet_replica.cc:330] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:20.847348 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 46d94c6f01544219a5602b3f06cb35b7 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.847426 10022 raft_consensus.cc:2229] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:20.847469 10021 ts_tablet_manager.cc:1909] T 6f8f4639ce314b34b88f67315f3e4fa7 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:20.847698 10024 log.cc:1196] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/46d94c6f01544219a5602b3f06cb35b7
I20240503 14:13:20.847771 10022 raft_consensus.cc:2258] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.848485 10021 tablet_replica.cc:330] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:20.848578 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 6f8f4639ce314b34b88f67315f3e4fa7 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.848621 10022 ts_tablet_manager.cc:1875] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.848630 10020 ts_tablet_manager.cc:1888] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.848760 10021 raft_consensus.cc:2229] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.848794 10020 log.cc:1196] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/6f8f4639ce314b34b88f67315f3e4fa7
I20240503 14:13:20.848985 10021 raft_consensus.cc:2258] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.848315 10024 ts_tablet_manager.cc:1909] T 46d94c6f01544219a5602b3f06cb35b7 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:20.849300 10020 ts_tablet_manager.cc:1909] T 6f8f4639ce314b34b88f67315f3e4fa7 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:20.849730 10021 ts_tablet_manager.cc:1875] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.850181 10024 tablet_replica.cc:330] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:20.850319 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 46d94c6f01544219a5602b3f06cb35b7 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.850409 10024 raft_consensus.cc:2229] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:20.850500 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 6f8f4639ce314b34b88f67315f3e4fa7 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.850729 10024 raft_consensus.cc:2258] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.850883 10020 tablet_replica.cc:330] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:20.851191 10020 raft_consensus.cc:2229] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:20.851542 10020 raft_consensus.cc:2258] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:20.851717 10024 ts_tablet_manager.cc:1875] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.852358 10020 ts_tablet_manager.cc:1875] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:20.855479 10022 ts_tablet_manager.cc:1888] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.855700 10022 log.cc:1196] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/8d111a5f08914420a7a429987c9699e9
I20240503 14:13:20.856189 10022 ts_tablet_manager.cc:1909] T 8d111a5f08914420a7a429987c9699e9 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:20.857196 10021 ts_tablet_manager.cc:1888] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.857411 10021 log.cc:1196] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/8d111a5f08914420a7a429987c9699e9
I20240503 14:13:20.857501 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 8d111a5f08914420a7a429987c9699e9 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.858400 10021 ts_tablet_manager.cc:1909] T 8d111a5f08914420a7a429987c9699e9 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:20.859483 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 8d111a5f08914420a7a429987c9699e9 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.860689 10020 ts_tablet_manager.cc:1888] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.860838 10020 log.cc:1196] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/46d94c6f01544219a5602b3f06cb35b7
I20240503 14:13:20.860924 10024 ts_tablet_manager.cc:1888] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:20.861125 10024 log.cc:1196] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/6f8f4639ce314b34b88f67315f3e4fa7
I20240503 14:13:20.861464 10020 ts_tablet_manager.cc:1909] T 46d94c6f01544219a5602b3f06cb35b7 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:20.861539 10024 ts_tablet_manager.cc:1909] T 6f8f4639ce314b34b88f67315f3e4fa7 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:20.862454 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 46d94c6f01544219a5602b3f06cb35b7 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.862534 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 6f8f4639ce314b34b88f67315f3e4fa7 (table rebalance_test_table_8 [id=22e4c12668aa48f6b48f7e24be68b2c1]) successfully deleted
I20240503 14:13:20.884027 10051 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.888973 9997 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:20.921166 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:20.930070 10247 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:20.930589 10247 tablet.cc:1620] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:20.932439 10251 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:20.933125 10251 tablet.cc:1620] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:20.944623 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=5c958d6ebb404e4aa2e19925dac21794] alter complete (version 2)
I20240503 14:13:20.945425 10248 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:20.945899 10248 tablet.cc:1620] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:21.004225 10042 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.010535 10220 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.037127 9779 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:21.052662 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:21.055773 6910 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:13:21.062789 10247 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:21.063625 10248 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:21.073076 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=5c958d6ebb404e4aa2e19925dac21794] alter complete (version 3)
I20240503 14:13:21.083806 10251 tablet.cc:1722] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:21.093534 7039 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 14:13:21.093004 6657 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:21.099659 9512 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 14:13:21.114550 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 62a55d0a1d7c4d7aba6635de1994e60b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.115043 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a55d0a1d7c4d7aba6635de1994e60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.115778 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 62a55d0a1d7c4d7aba6635de1994e60b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.116400 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a55d0a1d7c4d7aba6635de1994e60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.121101 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 412af3d40608416591ed3bf3bc450d67 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.121599 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412af3d40608416591ed3bf3bc450d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.124848 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 412af3d40608416591ed3bf3bc450d67 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.125566 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412af3d40608416591ed3bf3bc450d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.128054 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5410b22c57427d96a32fc9957cea8e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.128502 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5410b22c57427d96a32fc9957cea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.129539 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 62a55d0a1d7c4d7aba6635de1994e60b (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.129981 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a55d0a1d7c4d7aba6635de1994e60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.132038 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5410b22c57427d96a32fc9957cea8e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.132566 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5410b22c57427d96a32fc9957cea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.135002 10301 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:21.137161 10301 tablet_bootstrap.cc:654] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.137110 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 7e5410b22c57427d96a32fc9957cea8e (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.137578 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5410b22c57427d96a32fc9957cea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.144529 9910 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:21.164656 10304 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:21.174289 10304 tablet_bootstrap.cc:654] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.176259 10306 tablet_bootstrap.cc:492] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:21.177794 6657 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:21.178057 6657 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:21.178426 10306 tablet_bootstrap.cc:654] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.180876 10303 tablet_bootstrap.cc:492] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:21.181798 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 7840eb058d94456083c4b6ff99e9b38c
I20240503 14:13:21.182843 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 7840eb058d94456083c4b6ff99e9b38c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:21.183733 10303 tablet_bootstrap.cc:654] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.184538 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 7840eb058d94456083c4b6ff99e9b38c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:21.187655 10305 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:21.189855 10304 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:21.190070 10305 tablet_bootstrap.cc:654] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.190552 10304 ts_tablet_manager.cc:1367] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240503 14:13:21.191629 10304 raft_consensus.cc:348] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.193763 10304 raft_consensus.cc:374] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.194046 10304 raft_consensus.cc:729] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.194558 10304 consensus_queue.cc:260] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.204207 10309 tablet_replica.cc:330] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:21.204368 10308 tablet_replica.cc:330] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:21.190104 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 7840eb058d94456083c4b6ff99e9b38c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:21.204578 10309 raft_consensus.cc:2229] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.204849 10309 raft_consensus.cc:2258] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.207377 10308 raft_consensus.cc:2229] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.208230 10308 raft_consensus.cc:2258] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:21.208688 6736 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 138583us
I20240503 14:13:21.210532 10308 ts_tablet_manager.cc:1875] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.211458 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 412af3d40608416591ed3bf3bc450d67 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:21.211937 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412af3d40608416591ed3bf3bc450d67. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.217962 10309 ts_tablet_manager.cc:1875] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.221309 10306 tablet_bootstrap.cc:492] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:21.221598 10306 ts_tablet_manager.cc:1367] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.000s
I20240503 14:13:21.223043 10303 tablet_bootstrap.cc:492] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:21.223223 10303 ts_tablet_manager.cc:1367] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20240503 14:13:21.224292 10301 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:21.224534 10301 ts_tablet_manager.cc:1367] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.008s
I20240503 14:13:21.224759 10303 raft_consensus.cc:348] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.224992 10303 raft_consensus.cc:374] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.225109 10303 raft_consensus.cc:729] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.226032 10301 raft_consensus.cc:348] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.226310 10301 raft_consensus.cc:374] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.226456 10301 raft_consensus.cc:729] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.226768 10301 consensus_queue.cc:260] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.228251 10301 ts_tablet_manager.cc:1398] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:21.228974 10311 tablet_replica.cc:330] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:21.229231 10311 raft_consensus.cc:2229] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:21.230049 10311 raft_consensus.cc:2258] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.233902 10303 consensus_queue.cc:260] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.235044 10301 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:21.236882 10301 tablet_bootstrap.cc:654] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.237321 10306 raft_consensus.cc:348] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.237646 10306 raft_consensus.cc:374] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.237828 10306 raft_consensus.cc:729] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.238164 10303 ts_tablet_manager.cc:1398] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 14:13:21.238567 7171 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:21.239007 10306 consensus_queue.cc:260] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.241758 10306 ts_tablet_manager.cc:1398] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 14:13:21.242638 10201 raft_consensus.cc:482] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.242797 10201 raft_consensus.cc:504] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.243849 10201 leader_election.cc:288] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:21.244303 10306 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:21.247509 10306 tablet_bootstrap.cc:654] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.255558 10303 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:21.258644 10311 ts_tablet_manager.cc:1875] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.274837 10301 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:21.275148 10301 ts_tablet_manager.cc:1367] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20240503 14:13:21.276094 10301 raft_consensus.cc:348] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.283002 10308 ts_tablet_manager.cc:1888] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:21.283777 10308 log.cc:1196] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/7840eb058d94456083c4b6ff99e9b38c
I20240503 14:13:21.284982 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:21.286419 10308 ts_tablet_manager.cc:1909] T 7840eb058d94456083c4b6ff99e9b38c P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:21.284346 10305 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:21.287074 10305 ts_tablet_manager.cc:1367] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.100s user 0.005s sys 0.000s
I20240503 14:13:21.287922 10304 ts_tablet_manager.cc:1398] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.097s user 0.000s sys 0.002s
I20240503 14:13:21.288182 10305 raft_consensus.cc:348] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.288399 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 7840eb058d94456083c4b6ff99e9b38c (table rebalancer_dynamic_table_new_name [id=5c958d6ebb404e4aa2e19925dac21794]) successfully deleted
I20240503 14:13:21.288477 10305 raft_consensus.cc:374] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.288714 10305 raft_consensus.cc:729] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.288985 10305 consensus_queue.cc:260] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.294512 10305 ts_tablet_manager.cc:1398] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:21.297624 10311 ts_tablet_manager.cc:1888] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:21.297789 10311 log.cc:1196] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/7840eb058d94456083c4b6ff99e9b38c
I20240503 14:13:21.296202 10314 tablet_bootstrap.cc:492] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:21.298614 9644 tablet_service.cc:1467] Processing CreateTablet for tablet ec26d7c6a0744b01997b993daf3b4e14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4]), partition=
I20240503 14:13:21.298985 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec26d7c6a0744b01997b993daf3b4e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.301030 9511 tablet_service.cc:1467] Processing CreateTablet for tablet ec26d7c6a0744b01997b993daf3b4e14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4]), partition=
I20240503 14:13:21.304512 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec26d7c6a0744b01997b993daf3b4e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.306344 10311 ts_tablet_manager.cc:1909] T 7840eb058d94456083c4b6ff99e9b38c P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:21.307406 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 7840eb058d94456083c4b6ff99e9b38c (table rebalancer_dynamic_table_new_name [id=5c958d6ebb404e4aa2e19925dac21794]) successfully deleted
I20240503 14:13:21.308440 10306 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:21.308676 10306 ts_tablet_manager.cc:1367] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.000s
I20240503 14:13:21.308954 10301 raft_consensus.cc:374] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.309095 10301 raft_consensus.cc:729] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.309600 10306 raft_consensus.cc:348] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.309922 10306 raft_consensus.cc:374] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.310038 10306 raft_consensus.cc:729] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.310354 10306 consensus_queue.cc:260] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.310508 9778 tablet_service.cc:1467] Processing CreateTablet for tablet ec26d7c6a0744b01997b993daf3b4e14 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4]), partition=
I20240503 14:13:21.310915 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec26d7c6a0744b01997b993daf3b4e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:21.309396 10301 consensus_queue.cc:260] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.314137 10306 ts_tablet_manager.cc:1398] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:21.317474 10306 tablet_bootstrap.cc:492] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:21.318797 10301 ts_tablet_manager.cc:1398] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240503 14:13:21.319384 10306 tablet_bootstrap.cc:654] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.319810 10301 tablet_bootstrap.cc:492] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:21.320339 9644 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:21.321797 10301 tablet_bootstrap.cc:654] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.326262 10303 tablet_bootstrap.cc:654] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.330422 10306 tablet_bootstrap.cc:492] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:21.330672 10306 ts_tablet_manager.cc:1367] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 14:13:21.331701 10306 raft_consensus.cc:348] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.332083 10306 raft_consensus.cc:374] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.332254 10306 raft_consensus.cc:729] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.332513 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5410b22c57427d96a32fc9957cea8e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:21.332758 9533 raft_consensus.cc:2454] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:21.332628 10306 consensus_queue.cc:260] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.333576 7127 leader_election.cc:302] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:21.333921 10201 raft_consensus.cc:2781] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:21.334043 10201 raft_consensus.cc:482] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:21.334138 10201 raft_consensus.cc:3037] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.334987 10306 ts_tablet_manager.cc:1398] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:21.337164 10201 raft_consensus.cc:504] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.337819 10201 leader_election.cc:288] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:21.338883 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5410b22c57427d96a32fc9957cea8e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:21.339136 9533 raft_consensus.cc:3037] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.340198 9997 raft_consensus.cc:482] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.340412 9997 raft_consensus.cc:504] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.340708 6650 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 14:13:21.340917 6650 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 14:13:21.341342 9997 leader_election.cc:288] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:21.342098 9533 raft_consensus.cc:2454] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:21.345400 7127 leader_election.cc:302] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:21.345543 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet cef6534b85714da5a1a066b33b903f8a
I20240503 14:13:21.345715 10201 raft_consensus.cc:2781] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:21.346750 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet cef6534b85714da5a1a066b33b903f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:21.347115 10021 tablet_replica.cc:330] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:21.347415 10021 raft_consensus.cc:2229] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.347790 10021 raft_consensus.cc:2258] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.348614 10021 ts_tablet_manager.cc:1875] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.348707 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet cef6534b85714da5a1a066b33b903f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:21.348973 10020 tablet_replica.cc:330] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:21.349206 10020 raft_consensus.cc:2229] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.349439 10303 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:21.349463 10020 raft_consensus.cc:2258] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.349613 10303 ts_tablet_manager.cc:1367] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.094s user 0.001s sys 0.003s
I20240503 14:13:21.350131 10020 ts_tablet_manager.cc:1875] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.350526 10303 raft_consensus.cc:348] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.350782 10303 raft_consensus.cc:374] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.350855 10303 raft_consensus.cc:729] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.351814 10303 consensus_queue.cc:260] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.353722 10303 ts_tablet_manager.cc:1398] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:21.354070 10303 tablet_bootstrap.cc:492] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:21.356063 10303 tablet_bootstrap.cc:654] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.356057 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet a462f1d221e84693a523701a8b763b4e
I20240503 14:13:21.356822 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet cef6534b85714da5a1a066b33b903f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:21.356879 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet a462f1d221e84693a523701a8b763b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:21.357156 10022 tablet_replica.cc:330] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:21.357405 10022 raft_consensus.cc:2229] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:21.357750 10022 raft_consensus.cc:2258] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.358574 10022 ts_tablet_manager.cc:1875] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.358649 10020 ts_tablet_manager.cc:1888] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.358832 10020 log.cc:1196] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/cef6534b85714da5a1a066b33b903f8a
I20240503 14:13:21.359330 10020 ts_tablet_manager.cc:1909] T cef6534b85714da5a1a066b33b903f8a P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:21.360189 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet a462f1d221e84693a523701a8b763b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:21.360527 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet cef6534b85714da5a1a066b33b903f8a (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.361596 10201 raft_consensus.cc:686] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:21.361930 10201 consensus_queue.cc:237] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:21.364280 10020 tablet_replica.cc:330] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:21.364521 10020 raft_consensus.cc:2229] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.364806 10020 raft_consensus.cc:2258] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.365015 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 31f8e607d4fa40efb8d97d1c11ccf2c1
I20240503 14:13:21.366058 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:21.366402 10020 ts_tablet_manager.cc:1875] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.366662 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:21.368183 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet a462f1d221e84693a523701a8b763b4e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:21.368985 10022 ts_tablet_manager.cc:1888] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.369180 10022 log.cc:1196] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/cef6534b85714da5a1a066b33b903f8a
I20240503 14:13:21.370716 10021 ts_tablet_manager.cc:1888] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.371075 10021 log.cc:1196] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/cef6534b85714da5a1a066b33b903f8a
I20240503 14:13:21.371745 10021 ts_tablet_manager.cc:1909] T cef6534b85714da5a1a066b33b903f8a P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:21.375437 10250 raft_consensus.cc:482] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.375681 10250 raft_consensus.cc:504] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.376874 10250 leader_election.cc:288] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:21.379819 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet cef6534b85714da5a1a066b33b903f8a (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.385382 10309 ts_tablet_manager.cc:1888] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:21.385596 10309 log.cc:1196] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/7840eb058d94456083c4b6ff99e9b38c
I20240503 14:13:21.395988 10022 ts_tablet_manager.cc:1909] T cef6534b85714da5a1a066b33b903f8a P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:21.397372 10330 tablet_replica.cc:330] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:21.397619 10330 raft_consensus.cc:2229] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:21.397714 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet cef6534b85714da5a1a066b33b903f8a (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.398037 10330 raft_consensus.cc:2258] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.398192 10022 tablet_replica.cc:330] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:21.398449 10022 raft_consensus.cc:2229] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.398717 10022 raft_consensus.cc:2258] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.398823 10330 ts_tablet_manager.cc:1875] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.399477 10022 ts_tablet_manager.cc:1875] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.400811 10309 ts_tablet_manager.cc:1909] T 7840eb058d94456083c4b6ff99e9b38c P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:21.401751 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412af3d40608416591ed3bf3bc450d67" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:21.402115 9665 raft_consensus.cc:2454] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:21.402401 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 7840eb058d94456083c4b6ff99e9b38c (table rebalancer_dynamic_table_new_name [id=5c958d6ebb404e4aa2e19925dac21794]) successfully deleted
I20240503 14:13:21.404134 9733 leader_election.cc:302] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:21.404366 10021 tablet_replica.cc:330] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:21.404580 9997 raft_consensus.cc:2781] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:21.404733 9997 raft_consensus.cc:482] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:21.404870 9997 raft_consensus.cc:3037] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.404865 10021 raft_consensus.cc:2229] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.405222 10021 raft_consensus.cc:2258] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.406260 10021 ts_tablet_manager.cc:1875] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.408815 6778 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:13:21.408838 9997 raft_consensus.cc:504] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.409574 9997 leader_election.cc:288] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:21.410094 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:21.410485 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412af3d40608416591ed3bf3bc450d67" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:21.410847 9665 raft_consensus.cc:3037] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.413357 10314 tablet_bootstrap.cc:700] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13: Time spent opening tablet: real 0.109s user 0.002s sys 0.000s
I20240503 14:13:21.415565 9665 raft_consensus.cc:2454] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:21.417512 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5410b22c57427d96a32fc9957cea8e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:21.417737 9664 raft_consensus.cc:3037] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.417899 9733 leader_election.cc:302] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:21.418382 9997 raft_consensus.cc:2781] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:21.420795 9664 raft_consensus.cc:2454] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:21.421620 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e5410b22c57427d96a32fc9957cea8e" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:21.421928 9665 raft_consensus.cc:2362] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1. Re-sending same reply.
I20240503 14:13:21.424167 10301 tablet_bootstrap.cc:492] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:21.424337 10301 ts_tablet_manager.cc:1367] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.105s user 0.007s sys 0.000s
I20240503 14:13:21.425040 10314 tablet_bootstrap.cc:654] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:21.430421 9997 raft_consensus.cc:686] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:21.431034 9997 consensus_queue.cc:237] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
W20240503 14:13:21.431684 7237 tablet.cc:2369] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:21.440378 10314 tablet_bootstrap.cc:492] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:21.440418 10301 raft_consensus.cc:348] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.440688 10301 raft_consensus.cc:374] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.440783 10301 raft_consensus.cc:729] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.440901 10314 ts_tablet_manager.cc:1367] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.145s user 0.005s sys 0.000s
I20240503 14:13:21.446599 6650 catalog_manager.cc:5484] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:21.442080 10314 raft_consensus.cc:348] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.447886 10314 raft_consensus.cc:374] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.448046 10314 raft_consensus.cc:729] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.448422 10314 consensus_queue.cc:260] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:21.453841 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412af3d40608416591ed3bf3bc450d67" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:21.454192 6651 catalog_manager.cc:5484] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:21.456614 10314 ts_tablet_manager.cc:1398] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 14:13:21.463809 10021 ts_tablet_manager.cc:1888] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.465263 10020 ts_tablet_manager.cc:1888] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.465435 10020 log.cc:1196] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/a462f1d221e84693a523701a8b763b4e
I20240503 14:13:21.465873 10020 ts_tablet_manager.cc:1909] T a462f1d221e84693a523701a8b763b4e P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:21.466815 10022 ts_tablet_manager.cc:1888] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.466902 10020 tablet_replica.cc:330] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:21.467104 10020 raft_consensus.cc:2229] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:21.467108 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet a462f1d221e84693a523701a8b763b4e (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.467473 10020 raft_consensus.cc:2258] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.468616 10301 consensus_queue.cc:260] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.469013 10022 log.cc:1196] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/31f8e607d4fa40efb8d97d1c11ccf2c1
I20240503 14:13:21.469362 10301 ts_tablet_manager.cc:1398] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20240503 14:13:21.469578 10021 log.cc:1196] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/a462f1d221e84693a523701a8b763b4e
I20240503 14:13:21.469938 10022 ts_tablet_manager.cc:1909] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:21.470497 10021 ts_tablet_manager.cc:1909] T a462f1d221e84693a523701a8b763b4e P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:21.473698 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet a462f1d221e84693a523701a8b763b4e (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.474879 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.475029 10021 tablet_replica.cc:330] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:21.475373 10021 raft_consensus.cc:2229] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.479794 10021 raft_consensus.cc:2258] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.489189 10037 raft_consensus.cc:482] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.489399 10037 raft_consensus.cc:504] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.490134 10037 leader_election.cc:288] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:21.491056 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a55d0a1d7c4d7aba6635de1994e60b" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:21.491324 9929 raft_consensus.cc:2454] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:21.492134 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412af3d40608416591ed3bf3bc450d67" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:21.492384 6799 raft_consensus.cc:2454] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:21.492126 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412af3d40608416591ed3bf3bc450d67" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:21.493086 6798 raft_consensus.cc:3037] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.493669 10021 ts_tablet_manager.cc:1875] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.496121 9467 leader_election.cc:302] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:21.496182 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412af3d40608416591ed3bf3bc450d67" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:21.496789 9599 leader_election.cc:302] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5; no voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:21.497164 10037 raft_consensus.cc:2781] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:21.497290 10037 raft_consensus.cc:482] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:21.497318 10250 raft_consensus.cc:2726] T 412af3d40608416591ed3bf3bc450d67 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:21.497387 10037 raft_consensus.cc:3037] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:21.498672 6846 tablet.cc:2369] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:21.500059 6798 raft_consensus.cc:2454] T 412af3d40608416591ed3bf3bc450d67 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:21.501251 10037 raft_consensus.cc:504] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.502869 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a55d0a1d7c4d7aba6635de1994e60b" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:21.503093 9929 raft_consensus.cc:3037] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.505911 9929 raft_consensus.cc:2454] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:21.506590 9467 leader_election.cc:302] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:21.509330 10037 leader_election.cc:288] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:21.509774 10037 raft_consensus.cc:2781] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:21.510222 10037 raft_consensus.cc:686] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:21.510896 10037 consensus_queue.cc:237] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.513378 10338 raft_consensus.cc:482] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.513597 10338 raft_consensus.cc:504] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:21.513862 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a55d0a1d7c4d7aba6635de1994e60b" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:21.514125 9799 raft_consensus.cc:2454] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:21.514478 10338 leader_election.cc:288] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:21.514997 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a55d0a1d7c4d7aba6635de1994e60b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:21.515451 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a55d0a1d7c4d7aba6635de1994e60b" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:21.515708 9799 raft_consensus.cc:3037] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.516463 10021 ts_tablet_manager.cc:1888] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.516589 10021 log.cc:1196] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/31f8e607d4fa40efb8d97d1c11ccf2c1
I20240503 14:13:21.517025 10021 ts_tablet_manager.cc:1909] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:21.518232 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.518514 9799 raft_consensus.cc:2454] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:21.519829 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a55d0a1d7c4d7aba6635de1994e60b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:21.520778 9929 raft_consensus.cc:2379] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3efdbd3506274dba846ae40404f0572c in current term 1: Already voted for candidate 5b7e36bf354f480c90859cb8867b59cf in this term.
I20240503 14:13:21.521349 10303 tablet_bootstrap.cc:492] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:21.521502 10303 ts_tablet_manager.cc:1367] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.168s user 0.024s sys 0.024s
I20240503 14:13:21.522265 10303 raft_consensus.cc:348] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.522480 10303 raft_consensus.cc:374] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:21.522558 10303 raft_consensus.cc:729] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:21.522754 10303 consensus_queue.cc:260] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.523267 10303 ts_tablet_manager.cc:1398] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:21.523850 9733 leader_election.cc:302] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:21.525328 10338 raft_consensus.cc:2726] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:21.528713 10330 ts_tablet_manager.cc:1888] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.528863 10330 log.cc:1196] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/a462f1d221e84693a523701a8b763b4e
I20240503 14:13:21.529352 10330 ts_tablet_manager.cc:1909] T a462f1d221e84693a523701a8b763b4e P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:21.530449 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet a462f1d221e84693a523701a8b763b4e (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.538185 6651 catalog_manager.cc:5484] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:21.544210 10020 ts_tablet_manager.cc:1875] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.550136 10020 ts_tablet_manager.cc:1888] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.550446 10020 log.cc:1196] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/31f8e607d4fa40efb8d97d1c11ccf2c1
I20240503 14:13:21.551045 10020 ts_tablet_manager.cc:1909] T 31f8e607d4fa40efb8d97d1c11ccf2c1 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:21.552325 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 31f8e607d4fa40efb8d97d1c11ccf2c1 (table rebalance_test_table_10 [id=4beda38afa2a41dbb261be14a79cbe37]) successfully deleted
I20240503 14:13:21.609529 10037 raft_consensus.cc:482] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:21.609772 10037 raft_consensus.cc:504] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.610507 10037 leader_election.cc:288] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:21.611105 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec26d7c6a0744b01997b993daf3b4e14" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:21.611338 9799 raft_consensus.cc:2454] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:21.611368 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec26d7c6a0744b01997b993daf3b4e14" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:21.611646 9665 raft_consensus.cc:2454] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:21.612012 9468 leader_election.cc:302] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:21.612345 10037 raft_consensus.cc:2781] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:21.612433 10037 raft_consensus.cc:482] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:21.612529 10037 raft_consensus.cc:3037] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.615443 10037 raft_consensus.cc:504] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.616784 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec26d7c6a0744b01997b993daf3b4e14" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:21.617017 9799 raft_consensus.cc:3037] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.617208 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec26d7c6a0744b01997b993daf3b4e14" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:21.617431 9665 raft_consensus.cc:3037] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:21.620213 9799 raft_consensus.cc:2454] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:21.620741 9665 raft_consensus.cc:2454] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:21.620832 9468 leader_election.cc:302] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:21.621189 10340 raft_consensus.cc:2781] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:21.622102 10037 leader_election.cc:288] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:21.633379 10340 raft_consensus.cc:686] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:21.636108 10340 consensus_queue.cc:237] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:21.646891 6650 catalog_manager.cc:5484] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:21.657991 7190 consensus_queue.cc:237] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:21.661337 7058 raft_consensus.cc:1261] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.661944 6799 raft_consensus.cc:1261] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.662465 10201 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:21.662919 10327 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:21.667853 7126 consensus_peers.cc:485] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c95eaefadbff4511b610ab85cdfeb909. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:21.667325 10201 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:21.668262 7058 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:21.668535 6799 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:21.673223 6650 catalog_manager.cc:5484] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:21.696250 6799 consensus_queue.cc:237] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:21.700073 7190 raft_consensus.cc:1261] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.701057 9583 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:21.706569 6931 raft_consensus.cc:1261] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.708385 10296 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:21.724987 9583 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:21.734011 7190 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:21.735935 6658 catalog_manager.cc:5484] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:21.741108 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d5781b2310f4814aaebdcb715b7ddac. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:21.742676 6931 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:21.783363 7060 consensus_queue.cc:237] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:21.786290 7191 raft_consensus.cc:1261] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:21.786859 6995 consensus_peers.cc:485] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2a7661a6f5f34a668d5a998c7ce81276. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:21.786864 6799 raft_consensus.cc:1261] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.787444 10109 consensus_queue.cc:1035] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:21.787844 9715 consensus_queue.cc:1035] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:21.791620 10371 raft_consensus.cc:2932] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:21.792188 6799 raft_consensus.cc:2932] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:21.797616 7190 raft_consensus.cc:2932] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:21.798372 6657 catalog_manager.cc:5484] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:21.805230 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:21.816115 9799 raft_consensus.cc:1261] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:21.816344 9665 raft_consensus.cc:1261] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:21.816943 10342 consensus_queue.cc:1035] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.817644 10342 consensus_queue.cc:1035] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.835155 7060 consensus_queue.cc:237] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:21.838234 6658 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet ec26d7c6a0744b01997b993daf3b4e14 (table rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4]). Expected version 1 got 0
I20240503 14:13:21.838685 6799 raft_consensus.cc:1261] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.839946 6931 raft_consensus.cc:1261] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.840441 10371 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:21.840664 10338 consensus_queue.cc:1035] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.840793 10109 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:21.845299 6651 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet ec26d7c6a0744b01997b993daf3b4e14 (table rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4]). Expected version 1 got 0
I20240503 14:13:21.857872 10371 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:21.858448 6798 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:21.859490 6651 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet ec26d7c6a0744b01997b993daf3b4e14 (table rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4]). Expected version 1 got 0
I20240503 14:13:21.860165 10385 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:21.860765 10385 tablet.cc:1620] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:21.861135 10387 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:21.861773 10387 tablet.cc:1620] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:21.862922 10338 consensus_queue.cc:1035] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.869381 6651 catalog_manager.cc:5484] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:21.871098 6657 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 14:13:21.871389 6657 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 14:13:21.879163 10392 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:21.882339 10392 tablet.cc:1620] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:21.884316 6650 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet ec26d7c6a0744b01997b993daf3b4e14 (table rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4]). Expected version 1 got 0
I20240503 14:13:21.886842 6931 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:21.887703 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4] alter complete (version 1)
I20240503 14:13:21.888164 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet cc98f88f68af446696b49b1ddd9bde51
I20240503 14:13:21.888677 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet b045a23f3e584ee1a144645d93cf6982
I20240503 14:13:21.889149 6657 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 9d58cbd2efb746148f7deb34a38cd442
I20240503 14:13:21.890724 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet b045a23f3e584ee1a144645d93cf6982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:21.891156 10022 tablet_replica.cc:330] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:21.891371 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d58cbd2efb746148f7deb34a38cd442 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:21.891552 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet cc98f88f68af446696b49b1ddd9bde51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:21.891571 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet cc98f88f68af446696b49b1ddd9bde51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:21.891675 10330 tablet_replica.cc:330] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:21.891880 10330 raft_consensus.cc:2229] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.892153 10330 raft_consensus.cc:2258] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.892297 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet b045a23f3e584ee1a144645d93cf6982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:21.892556 10021 tablet_replica.cc:330] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:21.892781 10021 raft_consensus.cc:2229] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.892967 10330 ts_tablet_manager.cc:1875] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.892977 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet cc98f88f68af446696b49b1ddd9bde51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:21.893051 10021 raft_consensus.cc:2258] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.893752 10021 ts_tablet_manager.cc:1875] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.895756 10020 tablet_replica.cc:330] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:21.895793 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d58cbd2efb746148f7deb34a38cd442 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:21.895983 10020 raft_consensus.cc:2229] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.896250 10020 raft_consensus.cc:2258] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.896278 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet b045a23f3e584ee1a144645d93cf6982 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:21.896514 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d58cbd2efb746148f7deb34a38cd442 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:21 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:21.896999 10020 ts_tablet_manager.cc:1875] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.897504 10022 raft_consensus.cc:2229] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:21.897913 10022 raft_consensus.cc:2258] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.898697 10022 ts_tablet_manager.cc:1875] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.904920 10330 ts_tablet_manager.cc:1888] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.905208 10330 log.cc:1196] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/9d58cbd2efb746148f7deb34a38cd442
W20240503 14:13:21.906323 6997 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21e68127f4f94413b1be984a87bec6a8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:21.913779 10327 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.918669 10022 ts_tablet_manager.cc:1888] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.921694 10330 ts_tablet_manager.cc:1909] T 9d58cbd2efb746148f7deb34a38cd442 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:21.923710 10022 log.cc:1196] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/b045a23f3e584ee1a144645d93cf6982
I20240503 14:13:21.925889 10020 ts_tablet_manager.cc:1888] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.926535 10022 ts_tablet_manager.cc:1909] T b045a23f3e584ee1a144645d93cf6982 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:21.927176 10021 ts_tablet_manager.cc:1888] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.927482 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 9d58cbd2efb746148f7deb34a38cd442 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:21.928550 10021 log.cc:1196] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/b045a23f3e584ee1a144645d93cf6982
I20240503 14:13:21.928916 10020 log.cc:1196] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/cc98f88f68af446696b49b1ddd9bde51
I20240503 14:13:21.929553 10021 ts_tablet_manager.cc:1909] T b045a23f3e584ee1a144645d93cf6982 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:21.930248 10020 ts_tablet_manager.cc:1909] T cc98f88f68af446696b49b1ddd9bde51 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:21.931882 10330 tablet_replica.cc:330] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:21.932155 10330 raft_consensus.cc:2229] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.932564 10022 tablet_replica.cc:330] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:21.932724 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet b045a23f3e584ee1a144645d93cf6982 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:21.933046 10022 raft_consensus.cc:2229] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:21.933183 10330 raft_consensus.cc:2258] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.933781 10327 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.934840 10022 raft_consensus.cc:2258] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.935457 10330 ts_tablet_manager.cc:1875] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.935866 7190 consensus_queue.cc:237] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:21.943807 10022 ts_tablet_manager.cc:1875] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.959513 10021 tablet_replica.cc:330] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:21.959617 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet b045a23f3e584ee1a144645d93cf6982 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:21.960176 10020 tablet_replica.cc:330] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:21.960223 10022 ts_tablet_manager.cc:1888] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.960268 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet cc98f88f68af446696b49b1ddd9bde51 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:21.960222 10342 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.960388 10022 log.cc:1196] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/cc98f88f68af446696b49b1ddd9bde51
I20240503 14:13:21.960534 10021 raft_consensus.cc:2229] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:21.960794 10021 raft_consensus.cc:2258] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.960840 10022 ts_tablet_manager.cc:1909] T cc98f88f68af446696b49b1ddd9bde51 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:21.961092 10020 raft_consensus.cc:2229] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.961313 10020 raft_consensus.cc:2258] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.961961 10020 ts_tablet_manager.cc:1875] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.963160 10330 ts_tablet_manager.cc:1888] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.963320 10330 log.cc:1196] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/cc98f88f68af446696b49b1ddd9bde51
I20240503 14:13:21.963871 10330 ts_tablet_manager.cc:1909] T cc98f88f68af446696b49b1ddd9bde51 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:21.964215 10021 ts_tablet_manager.cc:1875] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.965626 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet cc98f88f68af446696b49b1ddd9bde51 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:21.966843 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet cc98f88f68af446696b49b1ddd9bde51 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:21.970048 10020 ts_tablet_manager.cc:1888] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.970189 10020 log.cc:1196] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/b045a23f3e584ee1a144645d93cf6982
I20240503 14:13:21.970656 10020 ts_tablet_manager.cc:1909] T b045a23f3e584ee1a144645d93cf6982 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:21.971570 10020 tablet_replica.cc:330] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:21.971765 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet b045a23f3e584ee1a144645d93cf6982 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:21.971941 10021 ts_tablet_manager.cc:1888] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:21.971879 10020 raft_consensus.cc:2229] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:21.972115 10021 log.cc:1196] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/9d58cbd2efb746148f7deb34a38cd442
I20240503 14:13:21.972187 10020 raft_consensus.cc:2258] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:21.972617 10021 ts_tablet_manager.cc:1909] T 9d58cbd2efb746148f7deb34a38cd442 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:21.972950 10020 ts_tablet_manager.cc:1875] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:21.973727 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 9d58cbd2efb746148f7deb34a38cd442 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:21.975246 7060 raft_consensus.cc:1261] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.975903 6931 raft_consensus.cc:1261] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:21.976305 10350 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:21.977809 10201 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:21.979144 7126 consensus_peers.cc:485] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24090e6169dd44f8b47c5cd772934710. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:21.980269 10037 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:21.983763 10327 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:21.984390 6931 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:21.988628 7060 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:21.991617 6650 catalog_manager.cc:5484] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:22.001051 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:22.009413 10385 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:22.010056 10385 tablet.cc:1620] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:22.011577 10392 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:22.012233 10392 tablet.cc:1620] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:22.012790 10020 ts_tablet_manager.cc:1888] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:22.014982 10020 log.cc:1196] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/9d58cbd2efb746148f7deb34a38cd442
I20240503 14:13:22.015410 10020 ts_tablet_manager.cc:1909] T 9d58cbd2efb746148f7deb34a38cd442 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:22.016311 10387 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:22.016783 10387 tablet.cc:1620] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:22.020670 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 9d58cbd2efb746148f7deb34a38cd442 (table rebalance_test_table_12 [id=250600562892440abe53eca3546351d7]) successfully deleted
I20240503 14:13:22.025540 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=f9145a9590b044ddace2ae8a881324a4] alter complete (version 2)
I20240503 14:13:22.044466 6931 consensus_queue.cc:237] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:22.045855 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:22.047828 7190 raft_consensus.cc:1261] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:22.049136 9872 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.049656 6799 raft_consensus.cc:1261] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:22.050328 9872 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.057826 6910 tablet_service.cc:1467] Processing CreateTablet for tablet fd25a2c3a16e460fbf83c667887983b7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.058298 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd25a2c3a16e460fbf83c667887983b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.058467 9778 tablet_service.cc:1467] Processing CreateTablet for tablet fd25a2c3a16e460fbf83c667887983b7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.058835 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd25a2c3a16e460fbf83c667887983b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.060345 9511 tablet_service.cc:1467] Processing CreateTablet for tablet f217c08046f64be9ac120896c9a0c559 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.060760 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f217c08046f64be9ac120896c9a0c559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.064662 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 652f2bf72ec9462384afcf9854509767 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.064669 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 652f2bf72ec9462384afcf9854509767 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.065089 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652f2bf72ec9462384afcf9854509767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.065083 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652f2bf72ec9462384afcf9854509767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.065186 9910 tablet_service.cc:1467] Processing CreateTablet for tablet fd25a2c3a16e460fbf83c667887983b7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.065708 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd25a2c3a16e460fbf83c667887983b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.067219 9644 tablet_service.cc:1467] Processing CreateTablet for tablet f217c08046f64be9ac120896c9a0c559 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.067651 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f217c08046f64be9ac120896c9a0c559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.072414 6780 tablet_service.cc:1467] Processing CreateTablet for tablet f217c08046f64be9ac120896c9a0c559 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.072353 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 652f2bf72ec9462384afcf9854509767 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:22.072914 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652f2bf72ec9462384afcf9854509767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.072937 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f217c08046f64be9ac120896c9a0c559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.087571 10420 tablet_bootstrap.cc:492] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
W20240503 14:13:22.088744 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20c4e7c22c904daebe97897320f6226e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:22.091416 10419 ts_tablet_manager.cc:902] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:22.094306 10420 tablet_bootstrap.cc:654] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.094579 9276 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:22.095332 6798 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:22.097957 10422 tablet_bootstrap.cc:492] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:22.101446 10422 tablet_bootstrap.cc:654] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.102167 6655 catalog_manager.cc:5484] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:22.122952 10422 tablet_bootstrap.cc:492] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:22.122494 7191 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:22.123937 10422 ts_tablet_manager.cc:1367] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.008s
I20240503 14:13:22.125327 10422 raft_consensus.cc:348] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.127429 10422 raft_consensus.cc:374] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.127786 10422 raft_consensus.cc:729] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.128365 10422 consensus_queue.cc:260] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.129815 10422 ts_tablet_manager.cc:1398] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:22.130828 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:22.147572 10385 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:22.155277 10387 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:22.161664 10392 tablet.cc:1722] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:22.173580 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=f9145a9590b044ddace2ae8a881324a4] alter complete (version 3)
I20240503 14:13:22.179950 10431 tablet_bootstrap.cc:492] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:22.181236 10419 tablet_copy_client.cc:308] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:22.182427 10431 tablet_bootstrap.cc:654] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.184001 10423 tablet_bootstrap.cc:492] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:22.186648 10423 tablet_bootstrap.cc:654] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.188320 10425 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:22.190699 10425 tablet_bootstrap.cc:654] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.192128 10429 tablet_bootstrap.cc:492] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:22.194371 10429 tablet_bootstrap.cc:654] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.199054 10421 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:22.199676 10420 tablet_bootstrap.cc:492] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:22.201123 10420 ts_tablet_manager.cc:1367] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.114s user 0.011s sys 0.011s
I20240503 14:13:22.201956 7060 consensus_queue.cc:237] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:22.204491 6931 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:22.204874 6995 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca522e5b44fd44289b4b161f27f71d20. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:22.205480 10371 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.205775 6798 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:22.206835 10420 raft_consensus.cc:348] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.207069 10420 raft_consensus.cc:374] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.207172 10420 raft_consensus.cc:729] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.207405 10420 consensus_queue.cc:260] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.207667 10371 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.209311 10371 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:22.212208 6931 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:22.216041 10420 ts_tablet_manager.cc:1398] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.010s user 0.000s sys 0.001s
I20240503 14:13:22.218717 6655 catalog_manager.cc:5484] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:22.219805 6798 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:22.222703 10420 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:22.223786 10431 tablet_bootstrap.cc:492] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:22.224046 10429 tablet_bootstrap.cc:492] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:22.225019 10423 tablet_bootstrap.cc:492] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:22.225263 10429 ts_tablet_manager.cc:1367] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240503 14:13:22.225489 10423 ts_tablet_manager.cc:1367] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20240503 14:13:22.225986 10431 ts_tablet_manager.cc:1367] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240503 14:13:22.226888 10429 raft_consensus.cc:348] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.227949 10423 raft_consensus.cc:348] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.229481 10421 tablet_bootstrap.cc:654] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.230305 10431 raft_consensus.cc:348] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.230587 10429 raft_consensus.cc:374] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.230718 10423 raft_consensus.cc:374] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.230844 10431 raft_consensus.cc:374] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.230958 10431 raft_consensus.cc:729] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.231083 10429 raft_consensus.cc:729] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.231518 10423 raft_consensus.cc:729] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.232021 10429 consensus_queue.cc:260] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.232818 10420 tablet_bootstrap.cc:654] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.232980 10431 consensus_queue.cc:260] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.232481 10423 consensus_queue.cc:260] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.233642 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet c95eaefadbff4511b610ab85cdfeb909 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:36385)
I20240503 14:13:22.233897 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet c95eaefadbff4511b610ab85cdfeb909 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:36385: session id = 3efdbd3506274dba846ae40404f0572c-c95eaefadbff4511b610ab85cdfeb909
I20240503 14:13:22.236776 10429 ts_tablet_manager.cc:1398] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:22.237294 10431 ts_tablet_manager.cc:1398] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:22.239171 10425 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:22.239398 10425 ts_tablet_manager.cc:1367] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240503 14:13:22.238992 7211 tablet_copy_source_session.cc:205] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:22.240397 10425 raft_consensus.cc:348] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.240687 10425 raft_consensus.cc:374] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.240833 10425 raft_consensus.cc:729] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.241142 10425 consensus_queue.cc:260] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.241753 10425 ts_tablet_manager.cc:1398] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:22.243129 10423 ts_tablet_manager.cc:1398] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240503 14:13:22.243479 10425 tablet_bootstrap.cc:492] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:22.243570 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95eaefadbff4511b610ab85cdfeb909. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.245254 10425 tablet_bootstrap.cc:654] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.256506 10425 tablet_bootstrap.cc:492] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:22.256772 10425 ts_tablet_manager.cc:1367] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20240503 14:13:22.258111 10425 raft_consensus.cc:348] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.259344 10425 raft_consensus.cc:374] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.259645 10425 raft_consensus.cc:729] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.259908 10419 tablet_copy_client.cc:783] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:22.260255 10419 tablet_copy_client.cc:647] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:22.260227 10425 consensus_queue.cc:260] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.264016 10419 tablet_copy_client.cc:523] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:22.264829 10425 ts_tablet_manager.cc:1398] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:13:22.272887 10421 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:22.273519 10421 ts_tablet_manager.cc:1367] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.011s
I20240503 14:13:22.273957 10420 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:22.274189 10420 ts_tablet_manager.cc:1367] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.053s user 0.003s sys 0.001s
I20240503 14:13:22.274470 10421 raft_consensus.cc:348] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.274751 10421 raft_consensus.cc:374] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.274886 10421 raft_consensus.cc:729] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.275031 10420 raft_consensus.cc:348] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.275198 10421 consensus_queue.cc:260] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.275372 10420 raft_consensus.cc:374] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.275513 10420 raft_consensus.cc:729] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.275849 10420 consensus_queue.cc:260] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.276093 10421 ts_tablet_manager.cc:1398] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 14:13:22.277338 10420 ts_tablet_manager.cc:1398] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:22.278314 6655 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:22.278537 6655 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240503 14:13:22.283903 6977 tablet.cc:2369] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 14:13:22.291709 6655 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet ec26d7c6a0744b01997b993daf3b4e14
I20240503 14:13:22.293427 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet ec26d7c6a0744b01997b993daf3b4e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:22.293967 9778 tablet_service.cc:1514] Processing DeleteTablet for tablet ec26d7c6a0744b01997b993daf3b4e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:22.295393 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet ec26d7c6a0744b01997b993daf3b4e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:22.307408 10459 tablet_replica.cc:330] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:22.308594 10456 ts_tablet_manager.cc:902] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:22.309274 10459 raft_consensus.cc:2229] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:22.309738 10419 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:22.310422 10459 raft_consensus.cc:2258] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.316174 10459 ts_tablet_manager.cc:1875] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.316388 10458 tablet_replica.cc:330] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:22.316583 10458 raft_consensus.cc:2229] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.316839 10458 raft_consensus.cc:2258] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.317724 10458 ts_tablet_manager.cc:1875] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.321650 6798 consensus_queue.cc:237] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:22.323621 10037 raft_consensus.cc:482] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.323782 10037 raft_consensus.cc:504] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.324529 10037 leader_election.cc:288] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:22.325589 10459 ts_tablet_manager.cc:1888] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:22.325863 10459 log.cc:1196] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/ec26d7c6a0744b01997b993daf3b4e14
I20240503 14:13:22.327322 10459 ts_tablet_manager.cc:1909] T ec26d7c6a0744b01997b993daf3b4e14 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:22.329928 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet ec26d7c6a0744b01997b993daf3b4e14 (table rebalancer_dynamic_table_new_name [id=f9145a9590b044ddace2ae8a881324a4]) successfully deleted
I20240503 14:13:22.331208 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f217c08046f64be9ac120896c9a0c559" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:22.331454 9665 raft_consensus.cc:2454] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:22.332134 9467 leader_election.cc:302] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:22.332532 10037 raft_consensus.cc:2781] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.332659 10037 raft_consensus.cc:482] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.332764 10037 raft_consensus.cc:3037] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.335548 10037 raft_consensus.cc:504] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.336630 7191 raft_consensus.cc:1261] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:22.337973 10037 leader_election.cc:288] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:22.338641 6929 raft_consensus.cc:1261] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:22.339643 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f217c08046f64be9ac120896c9a0c559" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:22.339910 9665 raft_consensus.cc:3037] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.340088 10363 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.347983 10296 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.356225 9665 raft_consensus.cc:2454] T f217c08046f64be9ac120896c9a0c559 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:22.357291 9467 leader_election.cc:302] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:22.357815 10037 raft_consensus.cc:2781] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.357960 10037 raft_consensus.cc:686] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:22.358249 10037 consensus_queue.cc:237] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
W20240503 14:13:22.360215 6735 consensus_peers.cc:485] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a98e955a2faf4fc889ec01bd148dd354. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:22.370033 10456 tablet_copy_client.cc:308] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:34101
I20240503 14:13:22.373247 10457 tablet_replica.cc:330] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:22.373531 6820 tablet_copy_service.cc:140] P f1a669e28f014fe287932acf56bebc13: Received BeginTabletCopySession request for tablet 2d5781b2310f4814aaebdcb715b7ddac from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:37325)
I20240503 14:13:22.373811 10457 raft_consensus.cc:2229] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.373800 6820 tablet_copy_service.cc:161] P f1a669e28f014fe287932acf56bebc13: Beginning new tablet copy session on tablet 2d5781b2310f4814aaebdcb715b7ddac from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:37325: session id = f03dc37fe3234033bc665bed4f1b51f5-2d5781b2310f4814aaebdcb715b7ddac
I20240503 14:13:22.373850 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f217c08046f64be9ac120896c9a0c559" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:22.374137 6799 raft_consensus.cc:2454] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:22.374182 10457 raft_consensus.cc:2258] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.374603 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f217c08046f64be9ac120896c9a0c559" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:22.374847 6799 raft_consensus.cc:3037] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.377794 6799 raft_consensus.cc:2454] T f217c08046f64be9ac120896c9a0c559 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:22.381493 6820 tablet_copy_source_session.cc:205] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:22.382850 10457 ts_tablet_manager.cc:1875] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.384023 10456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d5781b2310f4814aaebdcb715b7ddac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.385810 10458 ts_tablet_manager.cc:1888] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:22.385998 10458 log.cc:1196] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/ec26d7c6a0744b01997b993daf3b4e14
I20240503 14:13:22.385773 6658 catalog_manager.cc:5484] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:22.386719 10458 ts_tablet_manager.cc:1909] T ec26d7c6a0744b01997b993daf3b4e14 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:22.387315 10455 ts_tablet_manager.cc:902] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:22.388123 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet ec26d7c6a0744b01997b993daf3b4e14 (table rebalancer_dynamic_table_new_name [id=f9145a9590b044ddace2ae8a881324a4]) successfully deleted
I20240503 14:13:22.389088 10464 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:22.393071 6658 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 14:13:22.393240 10456 tablet_copy_client.cc:783] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:22.393325 6658 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 14:13:22.393543 10456 tablet_copy_client.cc:647] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:22.396749 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:22.397641 10473 ts_tablet_manager.cc:902] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:22.398046 6650 catalog_manager.cc:5484] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:22.400590 7190 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:22.394718 6929 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:22.412796 10456 tablet_copy_client.cc:523] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:22.417385 10456 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:22.435345 10473 tablet_copy_client.cc:308] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:22.437999 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 906158194aa342c99a3c351bfec164cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]), partition=
I20240503 14:13:22.438488 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906158194aa342c99a3c351bfec164cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.439307 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 906158194aa342c99a3c351bfec164cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]), partition=
I20240503 14:13:22.439746 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906158194aa342c99a3c351bfec164cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.442494 10457 ts_tablet_manager.cc:1888] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:22.443815 6658 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet f209ec165a034a1682000ba382ad5edb
I20240503 14:13:22.442638 10457 log.cc:1196] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/ec26d7c6a0744b01997b993daf3b4e14
I20240503 14:13:22.446223 10457 ts_tablet_manager.cc:1909] T ec26d7c6a0744b01997b993daf3b4e14 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:22.447410 10425 tablet_bootstrap.cc:492] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:22.449448 10425 tablet_bootstrap.cc:654] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.450527 10335 raft_consensus.cc:482] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.450690 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet ec26d7c6a0744b01997b993daf3b4e14 (table rebalancer_dynamic_table_new_name [id=f9145a9590b044ddace2ae8a881324a4]) successfully deleted
I20240503 14:13:22.450691 10335 raft_consensus.cc:504] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.451498 10335 leader_election.cc:288] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:22.456323 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 2a7661a6f5f34a668d5a998c7ce81276 from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:42011)
I20240503 14:13:22.456554 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 2a7661a6f5f34a668d5a998c7ce81276 from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:42011: session id = ea801601713a4d55aee0de9e3473e0bc-2a7661a6f5f34a668d5a998c7ce81276
I20240503 14:13:22.456586 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd25a2c3a16e460fbf83c667887983b7" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:22.456858 9929 raft_consensus.cc:2454] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:22.460685 7081 tablet_copy_source_session.cc:205] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:22.461491 10425 tablet_bootstrap.cc:492] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:22.461714 10425 ts_tablet_manager.cc:1367] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240503 14:13:22.462721 10425 raft_consensus.cc:348] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.463225 10425 raft_consensus.cc:374] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.463405 10425 raft_consensus.cc:729] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.464581 10486 ts_tablet_manager.cc:902] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:22.464175 10425 consensus_queue.cc:260] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.465169 10473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a7661a6f5f34a668d5a998c7ce81276. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.467285 10425 ts_tablet_manager.cc:1398] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:22.468899 9733 leader_election.cc:302] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:22.469821 10489 raft_consensus.cc:2781] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.469925 10489 raft_consensus.cc:482] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.470124 10489 raft_consensus.cc:3037] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.475306 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 906158194aa342c99a3c351bfec164cd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]), partition=
I20240503 14:13:22.476471 6658 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 4 replicas of tablet 2a7661a6f5f34a668d5a998c7ce81276
I20240503 14:13:22.477047 6658 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 9dfc52354bde44a886f6aaf3b9231f38
I20240503 14:13:22.477234 10489 raft_consensus.cc:504] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.478818 6778 tablet_service.cc:1514] Processing DeleteTablet for tablet f209ec165a034a1682000ba382ad5edb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:22.478946 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:22.479146 10020 tablet_replica.cc:330] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:22.479177 6777 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfc52354bde44a886f6aaf3b9231f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:22.479378 10020 raft_consensus.cc:2229] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:22.479780 10020 raft_consensus.cc:2258] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.480585 10020 ts_tablet_manager.cc:1875] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.480640 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd25a2c3a16e460fbf83c667887983b7" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:22.480945 9929 raft_consensus.cc:3037] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.483673 9929 raft_consensus.cc:2454] T fd25a2c3a16e460fbf83c667887983b7 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:22.484155 9733 leader_election.cc:302] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:22.484586 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet f209ec165a034a1682000ba382ad5edb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:22.484689 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:22.484916 7041 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfc52354bde44a886f6aaf3b9231f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:22.485136 10335 raft_consensus.cc:2781] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.485837 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet f209ec165a034a1682000ba382ad5edb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:22.486218 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfc52354bde44a886f6aaf3b9231f38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:22.487695 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:22.489647 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906158194aa342c99a3c351bfec164cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.492688 10422 tablet_bootstrap.cc:492] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:22.494782 10422 tablet_bootstrap.cc:654] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.495750 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd25a2c3a16e460fbf83c667887983b7" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:22.495954 6930 raft_consensus.cc:3037] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.497282 10489 leader_election.cc:288] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:22.498996 6930 raft_consensus.cc:2454] T fd25a2c3a16e460fbf83c667887983b7 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:22.495447 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd25a2c3a16e460fbf83c667887983b7" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:22.501376 10419 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:22.501969 10419 tablet_bootstrap.cc:492] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:22.502403 10419 ts_tablet_manager.cc:1367] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.193s user 0.046s sys 0.016s
I20240503 14:13:22.503401 10419 raft_consensus.cc:348] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:22.504312 10419 raft_consensus.cc:729] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:22.504634 10419 consensus_queue.cc:260] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:22.505614 10419 ts_tablet_manager.cc:1398] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:22.506577 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-c95eaefadbff4511b610ab85cdfeb909 received from {username='slave'} at 127.0.186.7:36385
I20240503 14:13:22.506770 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-c95eaefadbff4511b610ab85cdfeb909 on tablet c95eaefadbff4511b610ab85cdfeb909 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:22.514861 10020 ts_tablet_manager.cc:1888] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:22.515041 10020 log.cc:1196] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/f209ec165a034a1682000ba382ad5edb
I20240503 14:13:22.519273 10423 tablet_bootstrap.cc:492] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:22.521229 10473 tablet_copy_client.cc:783] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:22.522032 10473 tablet_copy_client.cc:647] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:22.526048 10423 tablet_bootstrap.cc:654] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:22.526840 10020 ts_tablet_manager.cc:1909] T f209ec165a034a1682000ba382ad5edb P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:22.528043 10350 raft_consensus.cc:482] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.528303 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet f209ec165a034a1682000ba382ad5edb (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.528247 10350 raft_consensus.cc:504] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.528744 10020 tablet_replica.cc:330] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:22.529023 10020 raft_consensus.cc:2229] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.529067 10350 leader_election.cc:288] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:22.529405 10020 raft_consensus.cc:2258] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.530012 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652f2bf72ec9462384afcf9854509767" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:22.530278 9799 raft_consensus.cc:2454] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:22.530803 10020 ts_tablet_manager.cc:1875] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.530853 7126 leader_election.cc:302] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:22.532176 10350 raft_consensus.cc:2781] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.532286 10350 raft_consensus.cc:482] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.532403 10350 raft_consensus.cc:3037] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.535118 10350 raft_consensus.cc:504] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.536080 10492 tablet_replica.cc:330] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:22.536383 10492 raft_consensus.cc:2229] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.536722 10492 raft_consensus.cc:2258] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.537178 10335 raft_consensus.cc:686] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:22.537482 10335 consensus_queue.cc:237] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:22.537621 10492 ts_tablet_manager.cc:1875] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.541615 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652f2bf72ec9462384afcf9854509767" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:22.541859 9799 raft_consensus.cc:3037] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.544062 10493 tablet_replica.cc:330] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:22.544370 10493 raft_consensus.cc:2229] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.544624 9799 raft_consensus.cc:2454] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:22.544782 10493 raft_consensus.cc:2258] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.545542 7126 leader_election.cc:302] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:22.545979 10201 raft_consensus.cc:2781] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.546072 10493 ts_tablet_manager.cc:1875] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 14:13:22.548259 9863 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 99083.6us
I20240503 14:13:22.549165 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:22.552922 6658 catalog_manager.cc:5484] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:22.554536 10350 leader_election.cc:288] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:22.558682 10422 tablet_bootstrap.cc:492] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:22.559379 10490 tablet_replica.cc:330] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:22.560643 10490 raft_consensus.cc:2229] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.562769 10422 ts_tablet_manager.cc:1367] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.001s
I20240503 14:13:22.563767 10490 raft_consensus.cc:2258] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.564429 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652f2bf72ec9462384afcf9854509767" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:22.564713 9665 raft_consensus.cc:2454] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:22.565034 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652f2bf72ec9462384afcf9854509767" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:22.565238 9664 raft_consensus.cc:3037] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.568573 10490 ts_tablet_manager.cc:1875] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.566037 10422 raft_consensus.cc:348] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.572777 10422 raft_consensus.cc:374] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:13:22.573328 6644 catalog_manager.cc:4810] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): delete failed for tablet 2a7661a6f5f34a668d5a998c7ce81276 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7661a6f5f34a668d5a998c7ce81276 already in progress: copying tablet
I20240503 14:13:22.572930 10422 raft_consensus.cc:729] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.573933 10422 consensus_queue.cc:260] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.574143 9664 raft_consensus.cc:2454] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:22.574713 10422 ts_tablet_manager.cc:1398] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:13:22.578553 10490 ts_tablet_manager.cc:1888] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:22.578809 10490 log.cc:1196] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/f209ec165a034a1682000ba382ad5edb
I20240503 14:13:22.579510 10490 ts_tablet_manager.cc:1909] T f209ec165a034a1682000ba382ad5edb P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:22.580809 10490 tablet_replica.cc:330] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:22.581094 10490 raft_consensus.cc:2229] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:22.581905 10490 raft_consensus.cc:2258] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.582981 10490 ts_tablet_manager.cc:1875] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.583470 10455 tablet_copy_client.cc:308] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:22.582953 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet f209ec165a034a1682000ba382ad5edb (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.589959 10492 ts_tablet_manager.cc:1888] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:22.590123 10492 log.cc:1196] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/f209ec165a034a1682000ba382ad5edb
I20240503 14:13:22.590436 10490 ts_tablet_manager.cc:1888] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:22.590566 10492 ts_tablet_manager.cc:1909] T f209ec165a034a1682000ba382ad5edb P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:22.590687 10490 log.cc:1196] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/2a7661a6f5f34a668d5a998c7ce81276
I20240503 14:13:22.591110 10490 ts_tablet_manager.cc:1909] T 2a7661a6f5f34a668d5a998c7ce81276 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:22.591650 10492 tablet_replica.cc:330] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:22.591665 10490 tablet_replica.cc:330] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:22.591866 10492 raft_consensus.cc:2229] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:22.591912 10490 raft_consensus.cc:2229] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.592223 10492 raft_consensus.cc:2258] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.592262 10490 raft_consensus.cc:2258] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.592934 10490 ts_tablet_manager.cc:1875] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.593582 10493 ts_tablet_manager.cc:1888] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:22.593739 10493 log.cc:1196] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/2a7661a6f5f34a668d5a998c7ce81276
I20240503 14:13:22.594099 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 2a7661a6f5f34a668d5a998c7ce81276 (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.594193 10493 ts_tablet_manager.cc:1909] T 2a7661a6f5f34a668d5a998c7ce81276 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:22.594935 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet f209ec165a034a1682000ba382ad5edb (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.595052 10020 ts_tablet_manager.cc:1888] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:22.595234 10020 log.cc:1196] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/2a7661a6f5f34a668d5a998c7ce81276
I20240503 14:13:22.595748 10020 ts_tablet_manager.cc:1909] T 2a7661a6f5f34a668d5a998c7ce81276 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:22.596859 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 2a7661a6f5f34a668d5a998c7ce81276 (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.597281 10020 tablet_replica.cc:330] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:22.597252 9799 raft_consensus.cc:1203] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:22.597518 10020 raft_consensus.cc:2229] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.597748 10020 raft_consensus.cc:2258] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.598453 10020 ts_tablet_manager.cc:1875] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.601469 10492 ts_tablet_manager.cc:1875] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.604311 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 21e68127f4f94413b1be984a87bec6a8 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:53967)
I20240503 14:13:22.604534 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 21e68127f4f94413b1be984a87bec6a8 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:53967: session id = 5b7e36bf354f480c90859cb8867b59cf-21e68127f4f94413b1be984a87bec6a8
I20240503 14:13:22.607476 7081 tablet_copy_source_session.cc:205] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:22.608319 10419 ts_tablet_manager.cc:902] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:22.608834 10473 tablet_copy_client.cc:523] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:22.611289 10455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21e68127f4f94413b1be984a87bec6a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.614826 10492 ts_tablet_manager.cc:1888] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:22.615247 10020 ts_tablet_manager.cc:1888] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:22.615449 10020 log.cc:1196] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/9dfc52354bde44a886f6aaf3b9231f38
I20240503 14:13:22.616004 10020 ts_tablet_manager.cc:1909] T 9dfc52354bde44a886f6aaf3b9231f38 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:22.617175 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 9dfc52354bde44a886f6aaf3b9231f38 (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.617658 10492 log.cc:1196] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/9dfc52354bde44a886f6aaf3b9231f38
I20240503 14:13:22.618151 10492 ts_tablet_manager.cc:1909] T 9dfc52354bde44a886f6aaf3b9231f38 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:22.619422 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 9dfc52354bde44a886f6aaf3b9231f38 (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.620677 10455 tablet_copy_client.cc:783] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:22.621104 10455 tablet_copy_client.cc:647] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:22.623983 10455 tablet_copy_client.cc:523] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:22.624450 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 2a7661a6f5f34a668d5a998c7ce81276 (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.627310 10473 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:22.628563 10464 raft_consensus.cc:482] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.628767 10464 raft_consensus.cc:504] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.629067 10455 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:22.629750 10464 leader_election.cc:288] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:22.631234 10490 ts_tablet_manager.cc:1888] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:22.631392 10490 log.cc:1196] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/9dfc52354bde44a886f6aaf3b9231f38
I20240503 14:13:22.631850 10490 ts_tablet_manager.cc:1909] T 9dfc52354bde44a886f6aaf3b9231f38 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:22.633926 10201 raft_consensus.cc:686] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:22.634425 10201 consensus_queue.cc:237] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:22.637055 10423 tablet_bootstrap.cc:492] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:22.637300 10423 ts_tablet_manager.cc:1367] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.118s user 0.005s sys 0.007s
I20240503 14:13:22.637276 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906158194aa342c99a3c351bfec164cd" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:22.637516 9664 raft_consensus.cc:2454] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:22.638075 6734 leader_election.cc:302] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:22.638479 10464 raft_consensus.cc:2781] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:22.638574 10464 raft_consensus.cc:482] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:22.638340 10423 raft_consensus.cc:348] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.638687 10464 raft_consensus.cc:3037] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.638720 10423 raft_consensus.cc:374] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:22.638808 10423 raft_consensus.cc:729] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:22.639093 10423 consensus_queue.cc:260] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.641207 10423 ts_tablet_manager.cc:1398] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:22.641360 10464 raft_consensus.cc:504] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.642948 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 9dfc52354bde44a886f6aaf3b9231f38 (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.643409 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906158194aa342c99a3c351bfec164cd" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:22.643668 9664 raft_consensus.cc:3037] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.648921 6930 consensus_queue.cc:237] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:22.655972 9664 raft_consensus.cc:2454] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:22.658001 10464 leader_election.cc:288] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:22.658812 6734 leader_election.cc:302] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:22.659539 10296 raft_consensus.cc:2781] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:22.659713 10296 raft_consensus.cc:686] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:22.660066 10296 consensus_queue.cc:237] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.667831 10486 tablet_copy_client.cc:308] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:22.695977 6799 raft_consensus.cc:1261] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:22.696738 9276 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.716382 9276 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:22.721323 6653 catalog_manager.cc:5484] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:22.722570 6798 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:22.735008 10419 tablet_copy_client.cc:308] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:22.735827 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 24090e6169dd44f8b47c5cd772934710 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:36385)
I20240503 14:13:22.736023 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 24090e6169dd44f8b47c5cd772934710 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:36385: session id = 3efdbd3506274dba846ae40404f0572c-24090e6169dd44f8b47c5cd772934710
I20240503 14:13:22.737799 6658 catalog_manager.cc:5484] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:22.739010 7211 tablet_copy_source_session.cc:205] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:22.744614 6658 catalog_manager.cc:5484] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:22.747551 10419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24090e6169dd44f8b47c5cd772934710. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.775070 7060 raft_consensus.cc:1261] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:22.775266 10419 tablet_copy_client.cc:783] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:22.775537 10419 tablet_copy_client.cc:647] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:22.783653 10419 tablet_copy_client.cc:523] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:22.784524 9276 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.793241 10419 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:22.799115 10456 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:22.799793 10456 tablet_bootstrap.cc:492] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:22.800196 10456 ts_tablet_manager.cc:1367] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.383s user 0.061s sys 0.005s
I20240503 14:13:22.801127 10456 raft_consensus.cc:348] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:22.801518 10456 raft_consensus.cc:729] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:22.801808 10456 consensus_queue.cc:260] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:22.803658 10456 ts_tablet_manager.cc:1398] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:22.804560 6820 tablet_copy_service.cc:342] P f1a669e28f014fe287932acf56bebc13: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-2d5781b2310f4814aaebdcb715b7ddac received from {username='slave'} at 127.0.186.6:37325
I20240503 14:13:22.804744 6820 tablet_copy_service.cc:433] P f1a669e28f014fe287932acf56bebc13: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-2d5781b2310f4814aaebdcb715b7ddac on tablet 2d5781b2310f4814aaebdcb715b7ddac with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:22.804958 10455 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:22.805451 10455 tablet_bootstrap.cc:492] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:22.805788 10455 ts_tablet_manager.cc:1367] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.177s user 0.054s sys 0.009s
I20240503 14:13:22.806859 10455 raft_consensus.cc:348] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:22.807159 10455 raft_consensus.cc:729] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:22.807415 10455 consensus_queue.cc:260] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:22.808633 10455 ts_tablet_manager.cc:1398] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:22.809442 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-21e68127f4f94413b1be984a87bec6a8 received from {username='slave'} at 127.0.186.5:53967
I20240503 14:13:22.809613 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-21e68127f4f94413b1be984a87bec6a8 on tablet 21e68127f4f94413b1be984a87bec6a8 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:22.809896 7060 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:22.821159 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 20c4e7c22c904daebe97897320f6226e from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:60021)
I20240503 14:13:22.821421 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 20c4e7c22c904daebe97897320f6226e from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:60021: session id = f03dc37fe3234033bc665bed4f1b51f5-20c4e7c22c904daebe97897320f6226e
I20240503 14:13:22.823817 10340 raft_consensus.cc:482] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:22.824020 10340 raft_consensus.cc:504] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:22.824721 10340 leader_election.cc:288] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:22.826380 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906158194aa342c99a3c351bfec164cd" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:22.826882 10340 consensus_queue.cc:1035] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.828084 6951 tablet_copy_source_session.cc:205] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:22.827546 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906158194aa342c99a3c351bfec164cd" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:22.829660 9664 raft_consensus.cc:2379] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b7e36bf354f480c90859cb8867b59cf in current term 1: Already voted for candidate f1a669e28f014fe287932acf56bebc13 in this term.
I20240503 14:13:22.830230 9467 leader_election.cc:302] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:22.830384 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20c4e7c22c904daebe97897320f6226e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:22.830953 10340 raft_consensus.cc:3037] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:22.834308 10340 raft_consensus.cc:2726] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:22.846184 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906158194aa342c99a3c351bfec164cd" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:22.846428 9533 raft_consensus.cc:2454] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:22.847851 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906158194aa342c99a3c351bfec164cd" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:22.850950 9533 raft_consensus.cc:2454] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:22.852608 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
W20240503 14:13:22.871764 6867 consensus_peers.cc:485] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9815af88a784ba7a50a43ca776ecd20. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:22.873760 10486 tablet_copy_client.cc:783] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:22.874110 10486 tablet_copy_client.cc:647] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:22.874444 10525 ts_tablet_manager.cc:902] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:22.876262 10340 consensus_queue.cc:1035] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.876775 10529 ts_tablet_manager.cc:902] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:22.878407 9664 raft_consensus.cc:1261] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:22.879082 10296 consensus_queue.cc:1035] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.899734 9533 raft_consensus.cc:1261] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:22.900570 10464 consensus_queue.cc:1035] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.907668 10473 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:22.908381 10473 tablet_bootstrap.cc:492] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:22.908493 10486 tablet_copy_client.cc:523] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:22.908866 10473 ts_tablet_manager.cc:1367] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.282s user 0.060s sys 0.014s
I20240503 14:13:22.909858 10473 raft_consensus.cc:348] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:22.910496 10473 raft_consensus.cc:729] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:22.910984 10473 consensus_queue.cc:260] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:22.913484 10486 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:22.917735 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54300
W20240503 14:13:22.918538 6644 catalog_manager.cc:4810] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): delete failed for tablet 2a7661a6f5f34a668d5a998c7ce81276 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2a7661a6f5f34a668d5a998c7ce81276 already in progress: copying tablet
I20240503 14:13:22.920053 10473 ts_tablet_manager.cc:1398] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 14:13:22.924034 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a7661a6f5f34a668d5a998c7ce81276 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:22.924396 10501 tablet_replica.cc:330] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:22.924600 10501 raft_consensus.cc:2229] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Raft consensus shutting down.
I20240503 14:13:22.924835 10501 raft_consensus.cc:2258] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Raft consensus is shut down!
I20240503 14:13:22.925825 6930 consensus_queue.cc:237] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:22.926234 10501 ts_tablet_manager.cc:1875] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.930559 10491 consensus_queue.cc:1035] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:22.939685 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-2a7661a6f5f34a668d5a998c7ce81276 received from {username='slave'} at 127.0.186.8:42011
I20240503 14:13:22.939850 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-2a7661a6f5f34a668d5a998c7ce81276 on tablet 2a7661a6f5f34a668d5a998c7ce81276 with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:22.938380 10501 ts_tablet_manager.cc:1888] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:22.943434 10501 log.cc:1196] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/2a7661a6f5f34a668d5a998c7ce81276
I20240503 14:13:22.944182 10501 ts_tablet_manager.cc:1909] T 2a7661a6f5f34a668d5a998c7ce81276 P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:22.945533 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 2a7661a6f5f34a668d5a998c7ce81276 (table rebalance_test_table_14 [id=6d47bfc6a80b4aa1a848bb34b985e322]) successfully deleted
I20240503 14:13:22.948199 7060 raft_consensus.cc:1261] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:22.948709 7190 raft_consensus.cc:1261] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:22.949000 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5738d4265f941519b90b07334d06af9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:22.949929 9276 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.952591 9276 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:22.960414 9872 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:22.961344 7060 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:22.970355 6653 catalog_manager.cc:5484] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:22.989590 6653 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 14:13:22.989856 6653 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 14:13:22.994822 6653 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 910d66fc2b224c3e9d5254d0816b31ad
I20240503 14:13:22.996469 6653 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet e1a371f2fa0a4d7dbc81385e2d8df293
I20240503 14:13:22.997048 6653 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet b98704f96ef64ccca952b039dd4f6f3b
I20240503 14:13:22.997236 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 910d66fc2b224c3e9d5254d0816b31ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:22.997560 10492 tablet_replica.cc:330] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:22.997754 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet b98704f96ef64ccca952b039dd4f6f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:22.997797 10492 raft_consensus.cc:2229] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.998059 10493 tablet_replica.cc:330] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:22.998075 10492 raft_consensus.cc:2258] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.998303 10493 raft_consensus.cc:2229] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:22.997757 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet e1a371f2fa0a4d7dbc81385e2d8df293 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:22.998557 10493 raft_consensus.cc:2258] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:22.998832 10492 ts_tablet_manager.cc:1875] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:22.999425 10493 ts_tablet_manager.cc:1875] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.000308 7041 tablet_service.cc:1514] Processing DeleteTablet for tablet b98704f96ef64ccca952b039dd4f6f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:23.000600 10490 tablet_replica.cc:330] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:23.000806 10490 raft_consensus.cc:2229] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.001008 10490 raft_consensus.cc:2258] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.001682 10490 ts_tablet_manager.cc:1875] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.003877 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 910d66fc2b224c3e9d5254d0816b31ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:23.005188 6777 tablet_service.cc:1514] Processing DeleteTablet for tablet e1a371f2fa0a4d7dbc81385e2d8df293 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:23.005381 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet b98704f96ef64ccca952b039dd4f6f3b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:23.005494 10020 tablet_replica.cc:330] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:23.005702 10020 raft_consensus.cc:2229] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.005928 10020 raft_consensus.cc:2258] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.005945 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet e1a371f2fa0a4d7dbc81385e2d8df293 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:23.006652 10020 ts_tablet_manager.cc:1875] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.009104 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 910d66fc2b224c3e9d5254d0816b31ad with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:22 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:23.011884 10490 ts_tablet_manager.cc:1888] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.012096 10490 log.cc:1196] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/b98704f96ef64ccca952b039dd4f6f3b
I20240503 14:13:23.012559 10490 ts_tablet_manager.cc:1909] T b98704f96ef64ccca952b039dd4f6f3b P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:23.013744 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet b98704f96ef64ccca952b039dd4f6f3b (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.014680 10020 ts_tablet_manager.cc:1888] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.016041 10492 ts_tablet_manager.cc:1888] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.016234 10493 ts_tablet_manager.cc:1888] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.016299 10020 log.cc:1196] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/e1a371f2fa0a4d7dbc81385e2d8df293
I20240503 14:13:23.016420 10490 tablet_replica.cc:330] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:23.016477 10493 log.cc:1196] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/b98704f96ef64ccca952b039dd4f6f3b
I20240503 14:13:23.016652 10490 raft_consensus.cc:2229] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:23.016959 10020 ts_tablet_manager.cc:1909] T e1a371f2fa0a4d7dbc81385e2d8df293 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:23.017024 10490 raft_consensus.cc:2258] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.017747 10490 ts_tablet_manager.cc:1875] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.017958 10492 log.cc:1196] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/910d66fc2b224c3e9d5254d0816b31ad
I20240503 14:13:23.018268 10493 ts_tablet_manager.cc:1909] T b98704f96ef64ccca952b039dd4f6f3b P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:23.018517 10492 ts_tablet_manager.cc:1909] T 910d66fc2b224c3e9d5254d0816b31ad P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:23.019170 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet e1a371f2fa0a4d7dbc81385e2d8df293 (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.019263 10493 tablet_replica.cc:330] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:23.019443 10493 raft_consensus.cc:2229] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.020148 10020 tablet_replica.cc:330] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:23.020347 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet b98704f96ef64ccca952b039dd4f6f3b (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.020700 10020 raft_consensus.cc:2229] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:23.021454 10020 raft_consensus.cc:2258] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.021869 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 910d66fc2b224c3e9d5254d0816b31ad (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.022347 10492 tablet_replica.cc:330] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:23.023365 10020 ts_tablet_manager.cc:1875] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.023661 10493 raft_consensus.cc:2258] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.023721 10492 raft_consensus.cc:2229] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:23.024188 10492 raft_consensus.cc:2258] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.025336 10493 ts_tablet_manager.cc:1875] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.025416 10492 ts_tablet_manager.cc:1875] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.033707 6658 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]). Expected version 1 got 0
I20240503 14:13:23.034531 10335 consensus_queue.cc:1035] T fd25a2c3a16e460fbf83c667887983b7 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:23.040577 6658 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]). Expected version 1 got 0
I20240503 14:13:23.043198 10492 ts_tablet_manager.cc:1888] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.046242 10525 tablet_copy_client.cc:308] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:23.047194 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet ca522e5b44fd44289b4b161f27f71d20 from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:42011)
I20240503 14:13:23.047402 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet ca522e5b44fd44289b4b161f27f71d20 from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:42011: session id = ea801601713a4d55aee0de9e3473e0bc-ca522e5b44fd44289b4b161f27f71d20
I20240503 14:13:23.050482 7081 tablet_copy_source_session.cc:205] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:23.052037 10525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca522e5b44fd44289b4b161f27f71d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.058701 10492 log.cc:1196] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/e1a371f2fa0a4d7dbc81385e2d8df293
I20240503 14:13:23.058944 10020 ts_tablet_manager.cc:1888] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.059289 10020 log.cc:1196] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/b98704f96ef64ccca952b039dd4f6f3b
I20240503 14:13:23.059921 10529 tablet_copy_client.cc:308] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:34101
I20240503 14:13:23.060492 10493 ts_tablet_manager.cc:1888] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.060884 6820 tablet_copy_service.cc:140] P f1a669e28f014fe287932acf56bebc13: Received BeginTabletCopySession request for tablet a98e955a2faf4fc889ec01bd148dd354 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:60115)
I20240503 14:13:23.061764 6820 tablet_copy_service.cc:161] P f1a669e28f014fe287932acf56bebc13: Beginning new tablet copy session on tablet a98e955a2faf4fc889ec01bd148dd354 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:60115: session id = 3efdbd3506274dba846ae40404f0572c-a98e955a2faf4fc889ec01bd148dd354
I20240503 14:13:23.062841 10540 mvcc.cc:205] Tried to move back new op lower bound from 7023597989379432448 to 7023597988498063360. Current Snapshot: MvccSnapshot[applied={T|T < 7023597989379432448}]
I20240503 14:13:23.065136 6820 tablet_copy_source_session.cc:205] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:23.065829 10350 raft_consensus.cc:1050] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:23.069849 10529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98e955a2faf4fc889ec01bd148dd354. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.070230 10419 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:23.070683 10419 tablet_bootstrap.cc:492] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:23.070693 10350 consensus_queue.cc:237] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:23.070956 10419 ts_tablet_manager.cc:1367] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.278s user 0.054s sys 0.020s
I20240503 14:13:23.071826 10419 raft_consensus.cc:348] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:23.072103 10419 raft_consensus.cc:729] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:23.072348 10419 consensus_queue.cc:260] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:23.073448 10563 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:23.074152 10563 tablet.cc:1620] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:23.078313 10529 tablet_copy_client.cc:783] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:23.078893 10020 ts_tablet_manager.cc:1909] T b98704f96ef64ccca952b039dd4f6f3b P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:23.079092 10492 ts_tablet_manager.cc:1909] T e1a371f2fa0a4d7dbc81385e2d8df293 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:23.080392 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet b98704f96ef64ccca952b039dd4f6f3b (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.080577 6658 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]). Expected version 1 got 0
I20240503 14:13:23.081779 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet e1a371f2fa0a4d7dbc81385e2d8df293 (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.082273 10020 tablet_replica.cc:330] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:23.082209 10558 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:23.082620 10020 raft_consensus.cc:2229] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.082924 10020 raft_consensus.cc:2258] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.083091 10558 tablet.cc:1620] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:23.083725 10020 ts_tablet_manager.cc:1875] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.085408 10525 tablet_copy_client.cc:783] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:23.085752 10525 tablet_copy_client.cc:647] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:23.086040 7058 raft_consensus.cc:1261] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.087743 10499 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.088970 6650 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]). Expected version 1 got 0
I20240503 14:13:23.090268 10525 tablet_copy_client.cc:523] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:23.092571 6650 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]). Expected version 1 got 0
I20240503 14:13:23.093818 6799 raft_consensus.cc:1261] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.094848 10499 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.098120 6650 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab]). Expected version 1 got 0
I20240503 14:13:23.085851 10557 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:23.100090 10557 tablet.cc:1620] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:23.082372 10419 ts_tablet_manager.cc:1398] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 14:13:23.103876 10529 tablet_copy_client.cc:647] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:23.104657 10490 ts_tablet_manager.cc:1888] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.104882 10490 log.cc:1196] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/910d66fc2b224c3e9d5254d0816b31ad
I20240503 14:13:23.105439 10490 ts_tablet_manager.cc:1909] T 910d66fc2b224c3e9d5254d0816b31ad P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:23.107403 10529 tablet_copy_client.cc:523] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:23.107445 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 910d66fc2b224c3e9d5254d0816b31ad (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.109140 10525 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:23.111265 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-24090e6169dd44f8b47c5cd772934710 received from {username='slave'} at 127.0.186.7:36385
I20240503 14:13:23.111435 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-24090e6169dd44f8b47c5cd772934710 on tablet 24090e6169dd44f8b47c5cd772934710 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:23.084748 9799 raft_consensus.cc:1261] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.113638 10350 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.103381 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab] alter complete (version 1)
I20240503 14:13:23.116984 7191 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:23.118958 10020 ts_tablet_manager.cc:1888] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.119108 10020 log.cc:1196] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/910d66fc2b224c3e9d5254d0816b31ad
I20240503 14:13:23.119637 10020 ts_tablet_manager.cc:1909] T 910d66fc2b224c3e9d5254d0816b31ad P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:23.121498 10529 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:23.123816 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 910d66fc2b224c3e9d5254d0816b31ad (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.129758 10493 log.cc:1196] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/e1a371f2fa0a4d7dbc81385e2d8df293
I20240503 14:13:23.134546 10493 ts_tablet_manager.cc:1909] T e1a371f2fa0a4d7dbc81385e2d8df293 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:23.138418 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet e1a371f2fa0a4d7dbc81385e2d8df293 (table rebalance_test_table_16 [id=db584ee54c8046168e36d1ce8d1e1f9f]) successfully deleted
I20240503 14:13:23.142550 10201 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:23.143534 7060 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:23.147579 9799 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:23.152123 6658 catalog_manager.cc:5484] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:23.157120 10499 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:23.157037 6798 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:23.170583 7191 consensus_queue.cc:237] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:23.172814 10350 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:23.173215 6799 raft_consensus.cc:1261] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:23.174742 10350 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:23.177295 9799 raft_consensus.cc:1261] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:23.180119 10201 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:23.181353 10350 consensus_queue.cc:1035] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:23.186715 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c95eaefadbff4511b610ab85cdfeb909 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:23.188552 9799 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:23.190799 6799 raft_consensus.cc:2932] T c95eaefadbff4511b610ab85cdfeb909 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:23.195837 6658 catalog_manager.cc:5484] T c95eaefadbff4511b610ab85cdfeb909 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:23.209964 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet c95eaefadbff4511b610ab85cdfeb909 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:23.210366 10490 tablet_replica.cc:330] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:23.210628 10490 raft_consensus.cc:2229] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.210886 10490 raft_consensus.cc:2258] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.212169 10490 ts_tablet_manager.cc:1875] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:23.217940 10490 ts_tablet_manager.cc:1888] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:23.218163 10490 log.cc:1196] T c95eaefadbff4511b610ab85cdfeb909 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/c95eaefadbff4511b610ab85cdfeb909
I20240503 14:13:23.220062 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet c95eaefadbff4511b610ab85cdfeb909 (table rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]) successfully deleted
I20240503 14:13:23.242113 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:23.249105 10529 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:23.249642 10529 tablet_bootstrap.cc:492] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:23.249965 10529 ts_tablet_manager.cc:1367] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.137s user 0.044s sys 0.017s
I20240503 14:13:23.250856 10529 raft_consensus.cc:348] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:23.251119 10529 raft_consensus.cc:729] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:23.251333 10529 consensus_queue.cc:260] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:23.251966 10529 ts_tablet_manager.cc:1398] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:23.252939 6820 tablet_copy_service.cc:342] P f1a669e28f014fe287932acf56bebc13: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-a98e955a2faf4fc889ec01bd148dd354 received from {username='slave'} at 127.0.186.7:60115
I20240503 14:13:23.253088 6820 tablet_copy_service.cc:433] P f1a669e28f014fe287932acf56bebc13: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-a98e955a2faf4fc889ec01bd148dd354 on tablet a98e955a2faf4fc889ec01bd148dd354 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:23.255235 10557 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:23.256762 10557 tablet.cc:1620] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:23.256853 10563 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:23.257330 10563 tablet.cc:1620] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:23.260900 10558 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:23.261412 10558 tablet.cc:1620] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:23.269384 9532 raft_consensus.cc:1203] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:23.273851 6655 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=a0d42af1be3743ffbee68f5cb99778ab] alter complete (version 2)
I20240503 14:13:23.293897 10455 ts_tablet_manager.cc:902] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:23.294729 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:23.296737 10486 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:23.297156 10486 tablet_bootstrap.cc:492] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:23.297443 10486 ts_tablet_manager.cc:1367] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.384s user 0.056s sys 0.007s
I20240503 14:13:23.299185 10525 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:23.299760 10525 tablet_bootstrap.cc:492] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:23.300110 10525 ts_tablet_manager.cc:1367] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.198s user 0.057s sys 0.001s
I20240503 14:13:23.301002 10525 raft_consensus.cc:348] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:23.301348 10525 raft_consensus.cc:729] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:23.301620 10525 consensus_queue.cc:260] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:23.302506 10486 raft_consensus.cc:348] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:23.311544 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 9f30640fc81e4aeead38e3f8659c056a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.311965 9512 tablet_service.cc:1467] Processing CreateTablet for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.312080 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f30640fc81e4aeead38e3f8659c056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.312377 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9641f7f5a3e4debb5aabc6b8a7497f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.312805 10525 ts_tablet_manager.cc:1398] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240503 14:13:23.313416 10486 raft_consensus.cc:729] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:23.313673 10486 consensus_queue.cc:260] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:23.315019 10486 ts_tablet_manager.cc:1398] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.017s user 0.003s sys 0.001s
I20240503 14:13:23.315694 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 7760fdb06dac4a46ab4452a67d7110aa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.315889 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-ca522e5b44fd44289b4b161f27f71d20 received from {username='slave'} at 127.0.186.8:42011
I20240503 14:13:23.316026 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-ca522e5b44fd44289b4b161f27f71d20 on tablet ca522e5b44fd44289b4b161f27f71d20 with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:23.316144 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7760fdb06dac4a46ab4452a67d7110aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.320063 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 7760fdb06dac4a46ab4452a67d7110aa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.320611 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7760fdb06dac4a46ab4452a67d7110aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.324151 7041 tablet_service.cc:1467] Processing CreateTablet for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.324748 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9641f7f5a3e4debb5aabc6b8a7497f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.328087 10587 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:23.328061 9779 tablet_service.cc:1467] Processing CreateTablet for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.328505 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9641f7f5a3e4debb5aabc6b8a7497f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.329608 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 9f30640fc81e4aeead38e3f8659c056a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.329864 10587 tablet_bootstrap.cc:654] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.330008 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f30640fc81e4aeead38e3f8659c056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.334990 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-20c4e7c22c904daebe97897320f6226e received from {username='slave'} at 127.0.186.6:60021
I20240503 14:13:23.335233 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-20c4e7c22c904daebe97897320f6226e on tablet 20c4e7c22c904daebe97897320f6226e with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:23.337172 10587 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:23.337381 10587 ts_tablet_manager.cc:1367] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:13:23.337456 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 7760fdb06dac4a46ab4452a67d7110aa (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.338032 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7760fdb06dac4a46ab4452a67d7110aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.339946 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 9f30640fc81e4aeead38e3f8659c056a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:23.340370 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f30640fc81e4aeead38e3f8659c056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.348067 10587 raft_consensus.cc:348] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.348372 10587 raft_consensus.cc:374] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.348489 10587 raft_consensus.cc:729] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.348721 10587 consensus_queue.cc:260] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.349407 10587 ts_tablet_manager.cc:1398] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:13:23.352331 9664 raft_consensus.cc:1203] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:23.376659 6653 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:23.390630 10557 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:23.399688 10558 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:23.416891 10486 ts_tablet_manager.cc:902] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:23.418648 10563 tablet.cc:1722] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:23.424605 9930 raft_consensus.cc:1203] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:23.431555 9665 raft_consensus.cc:1203] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:23.446491 9799 raft_consensus.cc:1203] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:23.457809 9799 raft_consensus.cc:1203] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:23.462196 10594 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:23.468134 10594 tablet_bootstrap.cc:654] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.475142 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=a0d42af1be3743ffbee68f5cb99778ab] alter complete (version 3)
I20240503 14:13:23.475848 10590 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:23.480355 10594 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:23.480679 10594 ts_tablet_manager.cc:1367] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20240503 14:13:23.481684 10594 raft_consensus.cc:348] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:23.481900 10590 tablet_bootstrap.cc:654] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.482021 10594 raft_consensus.cc:374] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.482193 10594 raft_consensus.cc:729] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.482547 10594 consensus_queue.cc:260] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:23.483299 10594 ts_tablet_manager.cc:1398] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:23.483666 10594 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:23.485705 10594 tablet_bootstrap.cc:654] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.490360 10590 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:23.491276 10590 ts_tablet_manager.cc:1367] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.002s
I20240503 14:13:23.494076 10590 raft_consensus.cc:348] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.494882 10590 raft_consensus.cc:374] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.495193 10590 raft_consensus.cc:729] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.495903 10590 consensus_queue.cc:260] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.498452 10590 ts_tablet_manager.cc:1398] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 14:13:23.499487 10590 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:23.500027 6655 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 14:13:23.500368 6655 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 14:13:23.502257 10590 tablet_bootstrap.cc:654] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.513353 10594 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:23.513624 10594 ts_tablet_manager.cc:1367] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.014s
I20240503 14:13:23.514482 10594 raft_consensus.cc:348] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:23.514791 10594 raft_consensus.cc:374] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.514950 10594 raft_consensus.cc:729] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.515312 10594 consensus_queue.cc:260] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:23.515945 10594 ts_tablet_manager.cc:1398] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:23.527071 10590 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:23.527402 10590 ts_tablet_manager.cc:1367] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.002s
I20240503 14:13:23.528151 6655 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet fdd011ee413741b0be51e718992c1f4a
I20240503 14:13:23.528497 10590 raft_consensus.cc:348] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:23.528911 10590 raft_consensus.cc:374] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.529044 10590 raft_consensus.cc:729] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.529433 10590 consensus_queue.cc:260] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:23.529812 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd011ee413741b0be51e718992c1f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:23.530130 10590 ts_tablet_manager.cc:1398] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:23.530824 10492 tablet_replica.cc:330] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:23.530831 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd011ee413741b0be51e718992c1f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:23.531096 10020 tablet_replica.cc:330] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:23.531066 10492 raft_consensus.cc:2229] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:23.531280 10020 raft_consensus.cc:2229] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.531453 10492 raft_consensus.cc:2258] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.531512 10020 raft_consensus.cc:2258] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.531654 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet fdd011ee413741b0be51e718992c1f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:23.531947 10493 tablet_replica.cc:330] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:23.532110 6655 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet e634f4f40a8c4bc1925498d384a30c01
I20240503 14:13:23.532192 10493 raft_consensus.cc:2229] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.532246 10020 ts_tablet_manager.cc:1875] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.532265 10492 ts_tablet_manager.cc:1875] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.532423 10493 raft_consensus.cc:2258] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.533221 10493 ts_tablet_manager.cc:1875] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.534608 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet e634f4f40a8c4bc1925498d384a30c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:23.535046 6655 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet c18ac0e1d6c94714a1021a9888319f3c
I20240503 14:13:23.535835 6779 tablet_service.cc:1514] Processing DeleteTablet for tablet c18ac0e1d6c94714a1021a9888319f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:23.535863 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet e634f4f40a8c4bc1925498d384a30c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:23.535995 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet e634f4f40a8c4bc1925498d384a30c01 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:23.536295 10490 tablet_replica.cc:330] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:23.536521 10490 raft_consensus.cc:2229] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:23.536864 10490 raft_consensus.cc:2258] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.537660 10490 ts_tablet_manager.cc:1875] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.539391 10020 ts_tablet_manager.cc:1888] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.539508 10020 log.cc:1196] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/fdd011ee413741b0be51e718992c1f4a
I20240503 14:13:23.539965 10020 ts_tablet_manager.cc:1909] T fdd011ee413741b0be51e718992c1f4a P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:23.540185 10493 ts_tablet_manager.cc:1888] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.540338 10493 log.cc:1196] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/fdd011ee413741b0be51e718992c1f4a
I20240503 14:13:23.540774 10493 ts_tablet_manager.cc:1909] T fdd011ee413741b0be51e718992c1f4a P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:23.540969 10020 tablet_replica.cc:330] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:23.541189 10020 raft_consensus.cc:2229] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:23.541236 10492 ts_tablet_manager.cc:1888] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.541357 10492 log.cc:1196] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/fdd011ee413741b0be51e718992c1f4a
I20240503 14:13:23.541565 10020 raft_consensus.cc:2258] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.541774 10493 tablet_replica.cc:330] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:23.541776 10492 ts_tablet_manager.cc:1909] T fdd011ee413741b0be51e718992c1f4a P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:23.542035 10493 raft_consensus.cc:2229] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.542317 10493 raft_consensus.cc:2258] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.542654 10590 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:23.542901 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet fdd011ee413741b0be51e718992c1f4a (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.542989 10493 ts_tablet_manager.cc:1875] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.543332 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet fdd011ee413741b0be51e718992c1f4a (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.543952 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet c18ac0e1d6c94714a1021a9888319f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:23.544284 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet fdd011ee413741b0be51e718992c1f4a (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.544471 10590 tablet_bootstrap.cc:654] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.544888 10020 ts_tablet_manager.cc:1875] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.550143 10490 ts_tablet_manager.cc:1888] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.550823 10490 log.cc:1196] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/e634f4f40a8c4bc1925498d384a30c01
I20240503 14:13:23.551412 10490 ts_tablet_manager.cc:1909] T e634f4f40a8c4bc1925498d384a30c01 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:23.552830 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet e634f4f40a8c4bc1925498d384a30c01 (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.555016 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet c18ac0e1d6c94714a1021a9888319f3c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:23.555181 10020 ts_tablet_manager.cc:1888] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.555394 10492 tablet_replica.cc:330] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:23.555699 10492 raft_consensus.cc:2229] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.556010 10492 raft_consensus.cc:2258] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.556933 10492 ts_tablet_manager.cc:1875] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.558326 10020 log.cc:1196] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/c18ac0e1d6c94714a1021a9888319f3c
I20240503 14:13:23.558821 10493 ts_tablet_manager.cc:1888] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.559473 10590 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:23.559777 10590 ts_tablet_manager.cc:1367] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.007s
I20240503 14:13:23.560956 10590 raft_consensus.cc:348] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:23.561277 10590 raft_consensus.cc:374] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.561425 10590 raft_consensus.cc:729] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.561748 10590 consensus_queue.cc:260] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:23.562472 10590 ts_tablet_manager.cc:1398] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:23.563148 10020 ts_tablet_manager.cc:1909] T c18ac0e1d6c94714a1021a9888319f3c P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:23.563570 10493 log.cc:1196] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/e634f4f40a8c4bc1925498d384a30c01
I20240503 14:13:23.564297 10020 tablet_replica.cc:330] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:23.564539 10020 raft_consensus.cc:2229] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.564630 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet c18ac0e1d6c94714a1021a9888319f3c (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.564786 10493 ts_tablet_manager.cc:1909] T e634f4f40a8c4bc1925498d384a30c01 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:23.564841 10020 raft_consensus.cc:2258] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.565622 10020 ts_tablet_manager.cc:1875] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.566838 10592 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:23.567716 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet e634f4f40a8c4bc1925498d384a30c01 (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.568845 10493 tablet_replica.cc:330] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:23.569096 10493 raft_consensus.cc:2229] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.569340 10493 raft_consensus.cc:2258] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.570138 10493 ts_tablet_manager.cc:1875] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.572834 10020 ts_tablet_manager.cc:1888] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.573050 10020 log.cc:1196] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/e634f4f40a8c4bc1925498d384a30c01
I20240503 14:13:23.573601 10020 ts_tablet_manager.cc:1909] T e634f4f40a8c4bc1925498d384a30c01 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:23.574935 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet e634f4f40a8c4bc1925498d384a30c01 (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.576640 10493 ts_tablet_manager.cc:1888] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.576814 10493 log.cc:1196] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/c18ac0e1d6c94714a1021a9888319f3c
I20240503 14:13:23.577142 10592 tablet_bootstrap.cc:654] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.577261 10493 ts_tablet_manager.cc:1909] T c18ac0e1d6c94714a1021a9888319f3c P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:23.578505 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet c18ac0e1d6c94714a1021a9888319f3c (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.578994 10492 ts_tablet_manager.cc:1888] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 14:13:23.579790 10492 log.cc:1196] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/c18ac0e1d6c94714a1021a9888319f3c
I20240503 14:13:23.579777 10486 tablet_copy_client.cc:308] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:23.580226 10492 ts_tablet_manager.cc:1909] T c18ac0e1d6c94714a1021a9888319f3c P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:23.580655 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet f5738d4265f941519b90b07334d06af9 from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:60021)
I20240503 14:13:23.580818 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet f5738d4265f941519b90b07334d06af9 from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:60021: session id = f03dc37fe3234033bc665bed4f1b51f5-f5738d4265f941519b90b07334d06af9
I20240503 14:13:23.581358 10455 tablet_copy_client.cc:308] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:23.582487 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet c18ac0e1d6c94714a1021a9888319f3c (table rebalance_test_table_18 [id=933d6c5622ea4054be06e3797c7378ea]) successfully deleted
I20240503 14:13:23.588079 6951 tablet_copy_source_session.cc:205] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:23.589948 6650 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:23.590499 6650 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:23.595423 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 906158194aa342c99a3c351bfec164cd
I20240503 14:13:23.596657 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 906158194aa342c99a3c351bfec164cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:23.598294 10486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5738d4265f941519b90b07334d06af9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.605942 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 906158194aa342c99a3c351bfec164cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:23.606384 10020 tablet_replica.cc:330] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:23.606691 10020 raft_consensus.cc:2229] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:23.607161 10020 raft_consensus.cc:2258] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.607411 9513 tablet_service.cc:1514] Processing DeleteTablet for tablet 906158194aa342c99a3c351bfec164cd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:23 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:23.608609 10020 ts_tablet_manager.cc:1875] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.625874 10611 tablet_replica.cc:330] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:23.626415 10611 raft_consensus.cc:2229] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.632277 10020 ts_tablet_manager.cc:1888] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:23.632444 10020 log.cc:1196] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/906158194aa342c99a3c351bfec164cd
I20240503 14:13:23.633184 10610 tablet_replica.cc:330] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:23.633555 10610 raft_consensus.cc:2229] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:23.634903 10610 raft_consensus.cc:2258] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.634053 10020 ts_tablet_manager.cc:1909] T 906158194aa342c99a3c351bfec164cd P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:23.639814 10545 raft_consensus.cc:482] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.640029 10545 raft_consensus.cc:504] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.626835 10611 raft_consensus.cc:2258] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:23.643841 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30640fc81e4aeead38e3f8659c056a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:23.644272 10545 leader_election.cc:288] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
W20240503 14:13:23.644745 6867 leader_election.cc:341] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.644830 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30640fc81e4aeead38e3f8659c056a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:23.645124 9532 raft_consensus.cc:2454] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:23.648159 6867 leader_election.cc:302] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:23.649057 10545 raft_consensus.cc:2781] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.649214 10545 raft_consensus.cc:482] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.649333 10545 raft_consensus.cc:3037] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.661109 10545 raft_consensus.cc:504] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.662971 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30640fc81e4aeead38e3f8659c056a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:23.663110 10545 leader_election.cc:288] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
W20240503 14:13:23.663789 6867 leader_election.cc:341] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.664217 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30640fc81e4aeead38e3f8659c056a" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:23.664515 9532 raft_consensus.cc:3037] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.665283 10611 ts_tablet_manager.cc:1875] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.678298 9532 raft_consensus.cc:2454] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:23.679157 6867 leader_election.cc:302] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:23.680122 10546 raft_consensus.cc:2781] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.680255 10546 raft_consensus.cc:686] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:23.680629 10546 consensus_queue.cc:237] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.699071 10610 ts_tablet_manager.cc:1875] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:23.700414 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table_new_name [id=a0d42af1be3743ffbee68f5cb99778ab]) successfully deleted
I20240503 14:13:23.702965 10486 tablet_copy_client.cc:783] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:23.704033 10486 tablet_copy_client.cc:647] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:23.707376 10486 tablet_copy_client.cc:523] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:23.716631 6653 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:23.728860 10491 raft_consensus.cc:482] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.729063 10491 raft_consensus.cc:504] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:23.731074 10491 leader_election.cc:288] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:23.737020 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7760fdb06dac4a46ab4452a67d7110aa" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:23.737494 9532 raft_consensus.cc:2454] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:23.738094 9735 leader_election.cc:302] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:23.738615 10491 raft_consensus.cc:2781] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.738731 10491 raft_consensus.cc:482] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.738868 10491 raft_consensus.cc:3037] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.745414 10611 ts_tablet_manager.cc:1888] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:23.746738 10491 raft_consensus.cc:504] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:23.748692 10491 leader_election.cc:288] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:23.715621 6655 catalog_manager.cc:5484] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:23.762984 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 0109f8f397f24bcabb6ebbf84a1139a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2]), partition=
I20240503 14:13:23.762967 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 0109f8f397f24bcabb6ebbf84a1139a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2]), partition=
I20240503 14:13:23.763339 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 0109f8f397f24bcabb6ebbf84a1139a3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2]), partition=
I20240503 14:13:23.763391 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0109f8f397f24bcabb6ebbf84a1139a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.763414 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0109f8f397f24bcabb6ebbf84a1139a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.763733 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0109f8f397f24bcabb6ebbf84a1139a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.774256 10371 raft_consensus.cc:1050] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:23.775445 10610 ts_tablet_manager.cc:1888] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:23.775578 10371 consensus_queue.cc:237] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:23.782675 10486 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:23.789798 10622 tablet_bootstrap.cc:492] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:23.790225 6930 raft_consensus.cc:1261] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.791342 9715 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.795444 6799 raft_consensus.cc:1261] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.796346 9715 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.804416 10620 tablet_bootstrap.cc:492] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:23.805830 10620 tablet_bootstrap.cc:654] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.799103 10622 tablet_bootstrap.cc:654] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.808076 10592 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:23.808293 10592 ts_tablet_manager.cc:1367] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.242s user 0.019s sys 0.022s
I20240503 14:13:23.808434 10371 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:23.809250 10592 raft_consensus.cc:348] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.809499 10592 raft_consensus.cc:374] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.809625 10592 raft_consensus.cc:729] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.809901 10592 consensus_queue.cc:260] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.811571 10592 ts_tablet_manager.cc:1398] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:23.811939 10592 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:23.814901 10592 tablet_bootstrap.cc:654] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.815570 6930 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:23.820019 10610 log.cc:1196] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/906158194aa342c99a3c351bfec164cd
I20240503 14:13:23.820727 10610 ts_tablet_manager.cc:1909] T 906158194aa342c99a3c351bfec164cd P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:23.823813 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table_new_name [id=a0d42af1be3743ffbee68f5cb99778ab]) successfully deleted
I20240503 14:13:23.825851 10621 tablet_bootstrap.cc:492] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:23.831203 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet f9815af88a784ba7a50a43ca776ecd20 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:46745)
I20240503 14:13:23.831393 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet f9815af88a784ba7a50a43ca776ecd20 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:46745: session id = 5b7e36bf354f480c90859cb8867b59cf-f9815af88a784ba7a50a43ca776ecd20
I20240503 14:13:23.834481 6951 tablet_copy_source_session.cc:205] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:23.836310 9715 raft_consensus.cc:482] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.748435 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7760fdb06dac4a46ab4452a67d7110aa" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:23.836508 9715 raft_consensus.cc:504] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:23.836687 9533 raft_consensus.cc:3037] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.837677 10620 tablet_bootstrap.cc:492] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:23.837896 10620 ts_tablet_manager.cc:1367] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20240503 14:13:23.837890 9715 leader_election.cc:288] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:23.838375 10621 tablet_bootstrap.cc:654] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:23.838848 10620 raft_consensus.cc:348] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.839521 10620 raft_consensus.cc:374] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.788443 9531 raft_consensus.cc:1261] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.836241 10455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9815af88a784ba7a50a43ca776ecd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:23.839926 10620 raft_consensus.cc:729] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.840490 10620 consensus_queue.cc:260] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.840852 10622 tablet_bootstrap.cc:492] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:23.841137 10622 ts_tablet_manager.cc:1367] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240503 14:13:23.841934 10620 ts_tablet_manager.cc:1398] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:23.842718 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30640fc81e4aeead38e3f8659c056a" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:23.843024 9531 raft_consensus.cc:2379] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09723f236d8044538d920a37b5d9aa8c in current term 1: Already voted for candidate 7186ed8510424fd597173eaf10c57b12 in this term.
I20240503 14:13:23.844520 9715 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.846583 10455 tablet_copy_client.cc:783] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:23.846825 10455 tablet_copy_client.cc:647] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:23.847759 9533 raft_consensus.cc:2454] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:23.848306 9735 leader_election.cc:302] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:23.848652 10491 raft_consensus.cc:2781] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.848798 10491 raft_consensus.cc:686] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:23.848732 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f30640fc81e4aeead38e3f8659c056a" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:23.849120 10491 consensus_queue.cc:237] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:23.851881 6995 leader_election.cc:302] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c; no voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:23.853878 10611 log.cc:1196] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/906158194aa342c99a3c351bfec164cd
I20240503 14:13:23.854555 10611 ts_tablet_manager.cc:1909] T 906158194aa342c99a3c351bfec164cd P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:23.854722 10455 tablet_copy_client.cc:523] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:23.856549 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 906158194aa342c99a3c351bfec164cd (table rebalancer_dynamic_table_new_name [id=a0d42af1be3743ffbee68f5cb99778ab]) successfully deleted
I20240503 14:13:23.858125 9715 raft_consensus.cc:3037] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.860810 10340 raft_consensus.cc:482] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.860986 10340 raft_consensus.cc:504] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:23.861838 10340 leader_election.cc:288] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:23.862682 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9641f7f5a3e4debb5aabc6b8a7497f6" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:23.863111 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9641f7f5a3e4debb5aabc6b8a7497f6" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:23.863379 9799 raft_consensus.cc:2454] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:23.864032 9468 leader_election.cc:302] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:23.864051 9715 raft_consensus.cc:2726] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:13:23.864792 9469 leader_election.cc:341] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.864794 6655 catalog_manager.cc:5484] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:23.865447 10340 raft_consensus.cc:2781] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.865561 10340 raft_consensus.cc:482] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.865633 10340 raft_consensus.cc:3037] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.865779 10622 raft_consensus.cc:348] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.866081 10622 raft_consensus.cc:374] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.866195 10622 raft_consensus.cc:729] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.866551 10622 consensus_queue.cc:260] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.868834 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7760fdb06dac4a46ab4452a67d7110aa" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
W20240503 14:13:23.869557 9735 leader_election.cc:341] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.869573 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7760fdb06dac4a46ab4452a67d7110aa" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:23.871287 10340 raft_consensus.cc:504] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
W20240503 14:13:23.872231 9735 leader_election.cc:341] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.872479 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9641f7f5a3e4debb5aabc6b8a7497f6" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:23.872691 9799 raft_consensus.cc:3037] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.873827 6799 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:23.877254 10622 ts_tablet_manager.cc:1398] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:23.877982 10514 raft_consensus.cc:1050] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:23.878985 10514 consensus_queue.cc:237] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:23.875502 9799 raft_consensus.cc:2454] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:23.880245 9468 leader_election.cc:302] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:23.891218 10340 leader_election.cc:288] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:23.892163 9664 raft_consensus.cc:1261] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.892345 7191 raft_consensus.cc:1261] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.892375 10340 raft_consensus.cc:2781] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.892488 6930 raft_consensus.cc:1261] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.892518 10340 raft_consensus.cc:686] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:23.892787 10340 consensus_queue.cc:237] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:23.892927 10296 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.894495 10464 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.895202 10464 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.904891 10574 raft_consensus.cc:482] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:23.905086 10574 raft_consensus.cc:504] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.906050 10574 leader_election.cc:288] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:23.908288 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0109f8f397f24bcabb6ebbf84a1139a3" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:23.908586 9665 raft_consensus.cc:2454] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:23.909677 7125 leader_election.cc:302] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:23.910156 10574 raft_consensus.cc:2781] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:23.910282 10574 raft_consensus.cc:482] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:23.910395 10574 raft_consensus.cc:3037] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.911211 10455 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:23.910933 9531 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:23.914240 10574 raft_consensus.cc:504] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.915982 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0109f8f397f24bcabb6ebbf84a1139a3" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
W20240503 14:13:23.916615 7125 leader_election.cc:341] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.916575 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0109f8f397f24bcabb6ebbf84a1139a3" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:23.916779 9665 raft_consensus.cc:3037] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:23.920364 9665 raft_consensus.cc:2454] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:23.921216 7125 leader_election.cc:302] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters: f1a669e28f014fe287932acf56bebc13
I20240503 14:13:23.921517 10201 raft_consensus.cc:2781] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:23.921653 10201 raft_consensus.cc:686] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:23.921300 10514 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:23.921900 10201 consensus_queue.cc:237] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.923820 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9641f7f5a3e4debb5aabc6b8a7497f6" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
W20240503 14:13:23.924443 9469 leader_election.cc:341] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:13:23.924685 6995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.1:34101, user_credentials={real_user=slave}} blocked reactor thread for 46866.4us
I20240503 14:13:23.928552 10296 raft_consensus.cc:1050] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:23.928743 9665 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:23.929332 10296 consensus_queue.cc:237] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:23.930223 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0109f8f397f24bcabb6ebbf84a1139a3" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:23.922556 6653 catalog_manager.cc:5484] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:23.931716 7125 leader_election.cc:341] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:23.904613 6658 catalog_manager.cc:5484] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:23.928427 6650 catalog_manager.cc:5484] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:23.940635 7191 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:23.942585 10574 leader_election.cc:288] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:23.945168 10621 tablet_bootstrap.cc:492] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:23.945340 10621 ts_tablet_manager.cc:1367] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.120s user 0.002s sys 0.004s
I20240503 14:13:23.945163 6930 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:23.948935 6653 catalog_manager.cc:5484] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:23.950680 9799 raft_consensus.cc:1261] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.951409 10464 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.958580 10621 raft_consensus.cc:348] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.958832 10621 raft_consensus.cc:374] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:23.958940 10621 raft_consensus.cc:729] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:23.959183 10621 consensus_queue.cc:260] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:23.959462 6799 consensus_queue.cc:237] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:23.959780 10621 ts_tablet_manager.cc:1398] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.014s user 0.001s sys 0.002s
I20240503 14:13:23.961295 7190 raft_consensus.cc:1261] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.965518 10464 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.966653 10371 raft_consensus.cc:1050] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:23.967832 10371 consensus_queue.cc:237] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:23.968483 10499 raft_consensus.cc:1050] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:23.968879 9665 raft_consensus.cc:1261] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:23.969280 10499 consensus_queue.cc:237] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:23.969491 10464 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:23.950851 6929 raft_consensus.cc:1261] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.975615 10545 raft_consensus.cc:1050] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:23.975471 10296 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:23.976758 10545 consensus_queue.cc:237] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:23.977902 10296 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:23.987289 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d5781b2310f4814aaebdcb715b7ddac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:23.994072 10455 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:23.994558 10455 tablet_bootstrap.cc:492] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:23.994846 10455 ts_tablet_manager.cc:1367] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.084s user 0.054s sys 0.005s
I20240503 14:13:23.995479 6653 catalog_manager.cc:5484] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:23.995728 10455 raft_consensus.cc:348] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:23.996023 10455 raft_consensus.cc:729] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:23.996273 10455 consensus_queue.cc:260] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:23.998054 9930 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.998116 6928 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:23.999205 9664 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.000922 9663 raft_consensus.cc:1261] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.000944 10455 ts_tablet_manager.cc:1398] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240503 14:13:24.001756 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d5781b2310f4814aaebdcb715b7ddac with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:24.002138 10493 tablet_replica.cc:330] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:24.002375 10493 raft_consensus.cc:2229] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:24.002647 10493 raft_consensus.cc:2258] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:24.003484 9276 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.004495 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-f9815af88a784ba7a50a43ca776ecd20 received from {username='slave'} at 127.0.186.5:46745
I20240503 14:13:24.004638 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-f9815af88a784ba7a50a43ca776ecd20 on tablet f9815af88a784ba7a50a43ca776ecd20 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:24.006561 9715 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.006922 10493 ts_tablet_manager.cc:1875] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:24.007342 9715 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.015831 6799 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.015642 10296 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.017153 6929 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.017570 6931 raft_consensus.cc:1261] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.021821 10464 consensus_queue.cc:1035] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.023627 9799 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.034430 7191 raft_consensus.cc:1261] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.036125 9276 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.038269 6653 catalog_manager.cc:5484] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.040925 10633 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.045734 10493 ts_tablet_manager.cc:1888] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:24.046725 10493 log.cc:1196] T 2d5781b2310f4814aaebdcb715b7ddac P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/2d5781b2310f4814aaebdcb715b7ddac
I20240503 14:13:24.053412 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 2d5781b2310f4814aaebdcb715b7ddac (table rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]) successfully deleted
I20240503 14:13:24.056424 10633 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:24.062829 9276 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.064760 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:24.064597 9929 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:24.071985 6650 catalog_manager.cc:5484] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:24.072171 7189 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.072572 6653 catalog_manager.cc:5484] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:24.073145 7191 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.080420 9664 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.082582 7060 raft_consensus.cc:1261] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.083647 10574 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.085780 9799 raft_consensus.cc:1261] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.087229 10201 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.087792 6930 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:24.096216 9663 raft_consensus.cc:1261] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:24.096897 10574 consensus_queue.cc:1035] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.097422 6798 raft_consensus.cc:3037] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.101688 6798 raft_consensus.cc:1261] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:24.105015 7058 consensus_queue.cc:237] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:24.108803 9929 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.109349 10371 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.109575 6930 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.110188 10633 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.112358 10634 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:24.112869 9929 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:24.113605 10663 consensus_queue.cc:1035] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.116680 10592 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:24.116883 10592 ts_tablet_manager.cc:1367] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.305s user 0.028s sys 0.046s
I20240503 14:13:24.117352 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca522e5b44fd44289b4b161f27f71d20 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:24.117798 10592 raft_consensus.cc:348] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:24.118172 10592 raft_consensus.cc:374] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.118252 10592 raft_consensus.cc:729] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.118492 10592 consensus_queue.cc:260] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:24.119128 10592 ts_tablet_manager.cc:1398] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:24.119417 10592 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:24.121003 10592 tablet_bootstrap.cc:654] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.120160 6657 catalog_manager.cc:5484] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 5 to 6, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:24.125566 6929 raft_consensus.cc:1261] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.130297 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet ca522e5b44fd44289b4b161f27f71d20 with delete_type TABLET_DATA_TOMBSTONED (TS f1a669e28f014fe287932acf56bebc13 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:24.130710 10020 tablet_replica.cc:330] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:24.130918 10020 raft_consensus.cc:2229] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:24.130928 10499 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.131201 10020 raft_consensus.cc:2258] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:24.132141 10020 ts_tablet_manager.cc:1875] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:24.144845 6931 raft_consensus.cc:2932] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:24.145395 10020 ts_tablet_manager.cc:1888] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240503 14:13:24.145552 10020 log.cc:1196] T ca522e5b44fd44289b4b161f27f71d20 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/ca522e5b44fd44289b4b161f27f71d20
I20240503 14:13:24.146521 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet ca522e5b44fd44289b4b161f27f71d20 (table rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]) successfully deleted
I20240503 14:13:24.150290 6799 consensus_queue.cc:237] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:24.152757 9798 raft_consensus.cc:1261] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.154008 7191 raft_consensus.cc:1261] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.154667 10464 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.155210 10657 consensus_queue.cc:1035] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.157407 10464 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.160892 9798 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.163242 7191 raft_consensus.cc:2932] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.169318 10574 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.169930 6657 catalog_manager.cc:5484] T a98e955a2faf4fc889ec01bd148dd354 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.174437 6929 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.175391 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet a98e955a2faf4fc889ec01bd148dd354 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:24.181598 6655 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 0109f8f397f24bcabb6ebbf84a1139a3 (table rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2]). Expected version 1 got 0
W20240503 14:13:24.184388 6734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.2:45285, user_credentials={real_user=slave}} blocked reactor thread for 163187us
I20240503 14:13:24.185387 7060 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.186295 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a98e955a2faf4fc889ec01bd148dd354 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:24.185948 6929 raft_consensus.cc:2932] T 2d5781b2310f4814aaebdcb715b7ddac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.186866 10545 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.188145 6654 catalog_manager.cc:5484] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.197719 6799 raft_consensus.cc:1261] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.199066 10545 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.205232 10672 tablet_replica.cc:330] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:24.206703 10672 raft_consensus.cc:2229] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:24.208597 10672 raft_consensus.cc:2258] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:24.208690 9798 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.210175 10672 ts_tablet_manager.cc:1875] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:24.213256 7191 consensus_queue.cc:237] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:24.211117 10545 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.216413 6931 raft_consensus.cc:1261] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.217043 10663 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.218636 6655 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 0109f8f397f24bcabb6ebbf84a1139a3 (table rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2]). Expected version 1 got 0
I20240503 14:13:24.219379 9533 raft_consensus.cc:1203] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:24.223701 6799 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.224519 10499 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.231055 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 24090e6169dd44f8b47c5cd772934710 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:24.234179 10672 ts_tablet_manager.cc:1888] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:24.238026 6655 catalog_manager.cc:5484] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:24.237701 6931 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.242635 10660 mvcc.cc:205] Tried to move back new op lower bound from 7023597994370523136 to 7023597993666588672. Current Snapshot: MvccSnapshot[applied={T|T < 7023597994370523136}]
I20240503 14:13:24.243455 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 24090e6169dd44f8b47c5cd772934710 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:24.250743 10672 log.cc:1196] T a98e955a2faf4fc889ec01bd148dd354 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/a98e955a2faf4fc889ec01bd148dd354
I20240503 14:13:24.252197 9797 raft_consensus.cc:1261] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.253428 10663 consensus_queue.cc:1035] T 24090e6169dd44f8b47c5cd772934710 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.253870 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet a98e955a2faf4fc889ec01bd148dd354 (table rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]) successfully deleted
I20240503 14:13:24.255259 9797 raft_consensus.cc:2932] T 24090e6169dd44f8b47c5cd772934710 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:24.259070 10678 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:24.259343 10679 tablet_replica.cc:330] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:24.259680 10679 raft_consensus.cc:2229] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:24.259804 10678 tablet.cc:1620] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:24.260082 10679 raft_consensus.cc:2258] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:24.265736 10671 log.cc:924] Time spent T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20240503 14:13:24.266692 6657 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 0109f8f397f24bcabb6ebbf84a1139a3 (table rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2]). Expected version 1 got 0
I20240503 14:13:24.269490 10676 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:24.270833 10676 tablet.cc:1620] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:24.275101 10679 ts_tablet_manager.cc:1875] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:24.283707 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2] alter complete (version 1)
I20240503 14:13:24.284124 10680 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:24.284862 10680 tablet.cc:1620] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:24.285423 6654 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 0109f8f397f24bcabb6ebbf84a1139a3 (table rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2]). Expected version 1 got 0
I20240503 14:13:24.298955 10679 ts_tablet_manager.cc:1888] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:24.299194 10679 log.cc:1196] T 24090e6169dd44f8b47c5cd772934710 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/24090e6169dd44f8b47c5cd772934710
I20240503 14:13:24.302596 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 24090e6169dd44f8b47c5cd772934710 (table rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]) successfully deleted
I20240503 14:13:24.304618 10632 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.311175 10592 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:24.311359 10592 ts_tablet_manager.cc:1367] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.192s user 0.007s sys 0.004s
I20240503 14:13:24.317831 10592 raft_consensus.cc:348] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.318118 10592 raft_consensus.cc:374] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.318220 10592 raft_consensus.cc:729] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.324102 10486 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:24.324992 7060 raft_consensus.cc:3037] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.325666 10592 consensus_queue.cc:260] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.332623 10637 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.336000 10592 ts_tablet_manager.cc:1398] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 14:13:24.342017 10486 tablet_bootstrap.cc:492] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:24.342415 10486 ts_tablet_manager.cc:1367] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.560s user 0.057s sys 0.010s
I20240503 14:13:24.346261 10486 raft_consensus.cc:348] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:24.346549 10486 raft_consensus.cc:729] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:24.348555 10486 consensus_queue.cc:260] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:24.351290 10486 ts_tablet_manager.cc:1398] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:24.352023 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-f5738d4265f941519b90b07334d06af9 received from {username='slave'} at 127.0.186.6:60021
I20240503 14:13:24.352228 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-f5738d4265f941519b90b07334d06af9 on tablet f5738d4265f941519b90b07334d06af9 with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:24.383013 10491 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:24.404191 7059 consensus_queue.cc:237] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:24.407745 6798 raft_consensus.cc:1261] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.408177 9533 raft_consensus.cc:1261] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.408877 10634 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.409073 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:24.409513 10371 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.412082 10371 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:24.416726 9533 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:24.418495 6800 raft_consensus.cc:2932] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:24.420286 10676 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:24.420835 10676 tablet.cc:1620] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:24.422255 10678 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:24.422717 10678 tablet.cc:1620] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:24.426141 6651 catalog_manager.cc:5484] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:24.429581 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21e68127f4f94413b1be984a87bec6a8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:24.431021 10680 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:24.431511 10680 tablet.cc:1620] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:24.434535 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 21e68127f4f94413b1be984a87bec6a8 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:24.434923 10672 tablet_replica.cc:330] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:24.435122 10672 raft_consensus.cc:2229] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:24.435424 10672 raft_consensus.cc:2258] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:24.437103 10672 ts_tablet_manager.cc:1875] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:24.442137 7059 raft_consensus.cc:3037] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.448527 10698 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240503 14:13:24.451220 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=4558b2809da749ed90d80b6914f412e2] alter complete (version 2)
I20240503 14:13:24.457190 9663 raft_consensus.cc:1203] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:24.476536 10672 ts_tablet_manager.cc:1888] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:24.476694 10672 log.cc:1196] T 21e68127f4f94413b1be984a87bec6a8 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/21e68127f4f94413b1be984a87bec6a8
I20240503 14:13:24.480041 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 21e68127f4f94413b1be984a87bec6a8 (table rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]) successfully deleted
I20240503 14:13:24.513991 7039 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:13:24.521605 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:24.534952 9779 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:13:24.541674 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 1949c165716e42cf89d542e6b9b7ae71 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.542223 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1949c165716e42cf89d542e6b9b7ae71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.546219 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 1949c165716e42cf89d542e6b9b7ae71 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.546717 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1949c165716e42cf89d542e6b9b7ae71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.548314 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 1949c165716e42cf89d542e6b9b7ae71 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.548772 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1949c165716e42cf89d542e6b9b7ae71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.553121 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 7255e647ced644ed9b76937e7808a147 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.553395 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 7255e647ced644ed9b76937e7808a147 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.553648 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7255e647ced644ed9b76937e7808a147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.553828 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7255e647ced644ed9b76937e7808a147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.557585 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 62d262e208a84c14bd296e0c44947813 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.558106 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d262e208a84c14bd296e0c44947813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.558183 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 7255e647ced644ed9b76937e7808a147 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.558578 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7255e647ced644ed9b76937e7808a147. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.560477 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 62d262e208a84c14bd296e0c44947813 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.560950 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d262e208a84c14bd296e0c44947813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.565832 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:24.570688 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 62d262e208a84c14bd296e0c44947813 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:24.571290 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d262e208a84c14bd296e0c44947813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:24.597678 6910 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 14:13:24.605648 9513 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 14:13:24.639825 10715 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:24.641721 10715 tablet_bootstrap.cc:654] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.653091 10719 tablet_bootstrap.cc:492] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:24.657936 10719 tablet_bootstrap.cc:654] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.658594 10718 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:24.664724 10716 tablet_bootstrap.cc:492] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:24.665220 10714 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:24.681588 10715 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:24.681839 10716 tablet_bootstrap.cc:654] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.681866 10715 ts_tablet_manager.cc:1367] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.004s
I20240503 14:13:24.682704 10714 tablet_bootstrap.cc:654] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.682751 10715 raft_consensus.cc:348] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.684036 10718 tablet_bootstrap.cc:654] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.686089 10721 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:24.674844 9910 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 14:13:24.697636 10715 raft_consensus.cc:374] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.697860 10715 raft_consensus.cc:729] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.698199 10715 consensus_queue.cc:260] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.698513 10676 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:24.699818 10715 ts_tablet_manager.cc:1398] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240503 14:13:24.700093 10715 tablet_bootstrap.cc:492] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:24.701767 10715 tablet_bootstrap.cc:654] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.708427 10721 tablet_bootstrap.cc:654] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.715648 10680 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:24.727324 10718 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:24.727576 10718 ts_tablet_manager.cc:1367] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20240503 14:13:24.728523 10718 raft_consensus.cc:348] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.728811 10718 raft_consensus.cc:374] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.728955 10718 raft_consensus.cc:729] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.729269 10718 consensus_queue.cc:260] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.736464 10718 ts_tablet_manager.cc:1398] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:24.736670 9872 raft_consensus.cc:1050] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:24.737480 9872 consensus_queue.cc:237] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:24.739533 10718 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:24.743458 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=4558b2809da749ed90d80b6914f412e2] alter complete (version 3)
I20240503 14:13:24.702390 10678 tablet.cc:1722] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:24.748504 7171 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 14:13:24.754834 10715 tablet_bootstrap.cc:492] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:24.757203 6929 consensus_queue.cc:237] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:24.758435 10715 ts_tablet_manager.cc:1367] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20240503 14:13:24.760335 10721 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:24.760407 9663 raft_consensus.cc:1261] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.761344 6798 raft_consensus.cc:1261] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:24.761848 10545 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.764181 10545 consensus_queue.cc:1035] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:24.765255 10545 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.766356 10721 ts_tablet_manager.cc:1367] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.080s user 0.004s sys 0.000s
I20240503 14:13:24.767315 10721 raft_consensus.cc:348] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.767534 10721 raft_consensus.cc:374] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.767648 10721 raft_consensus.cc:729] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.767895 10721 consensus_queue.cc:260] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.767838 9663 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.768800 10721 ts_tablet_manager.cc:1398] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:24.774014 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20c4e7c22c904daebe97897320f6226e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:24.778584 6650 catalog_manager.cc:5484] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:24.781513 10718 tablet_bootstrap.cc:654] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.780594 10715 raft_consensus.cc:348] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:24.785637 10715 raft_consensus.cc:374] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.785981 10715 raft_consensus.cc:729] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.789112 10715 consensus_queue.cc:260] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:24.792047 10715 ts_tablet_manager.cc:1398] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240503 14:13:24.798108 10716 tablet_bootstrap.cc:492] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:24.798369 10716 ts_tablet_manager.cc:1367] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.134s user 0.005s sys 0.000s
I20240503 14:13:24.799264 10716 raft_consensus.cc:348] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:24.799563 10716 raft_consensus.cc:374] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.799772 10716 raft_consensus.cc:729] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.800112 10716 consensus_queue.cc:260] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:24.801683 10716 ts_tablet_manager.cc:1398] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:24.802031 10716 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:24.804073 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 20c4e7c22c904daebe97897320f6226e with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:24.814185 10714 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:24.814507 10714 ts_tablet_manager.cc:1367] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.149s user 0.007s sys 0.000s
I20240503 14:13:24.835099 10716 tablet_bootstrap.cc:654] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:24.815456 10714 raft_consensus.cc:348] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.837363 10714 raft_consensus.cc:374] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.837538 10714 raft_consensus.cc:729] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.837911 10714 consensus_queue.cc:260] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.847294 6658 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:24.850502 6658 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:24.858565 6658 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 0109f8f397f24bcabb6ebbf84a1139a3
I20240503 14:13:24.859364 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 0109f8f397f24bcabb6ebbf84a1139a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:24 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:24.860297 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 0109f8f397f24bcabb6ebbf84a1139a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:24 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:24.866444 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0109f8f397f24bcabb6ebbf84a1139a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:24 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:24.879321 10714 ts_tablet_manager.cc:1398] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.065s user 0.002s sys 0.000s
I20240503 14:13:24.900817 10716 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:24.901103 10716 ts_tablet_manager.cc:1367] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.000s
I20240503 14:13:24.902140 10716 raft_consensus.cc:348] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.902482 10716 raft_consensus.cc:374] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.902581 10716 raft_consensus.cc:729] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.902885 10716 consensus_queue.cc:260] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.903743 10716 ts_tablet_manager.cc:1398] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:24.905750 10737 tablet_replica.cc:330] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:24.906025 10737 raft_consensus.cc:2229] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:24.906270 10737 raft_consensus.cc:2258] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:24.906737 10731 tablet_replica.cc:330] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:24.906981 10731 raft_consensus.cc:2229] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:24.907335 10731 raft_consensus.cc:2258] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:24.907346 10736 tablet_replica.cc:330] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:24.908038 10736 raft_consensus.cc:2229] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:24.908365 10736 raft_consensus.cc:2258] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:24.909854 10296 raft_consensus.cc:482] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:24.910032 10296 raft_consensus.cc:504] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.910727 10296 leader_election.cc:288] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:24.911520 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1949c165716e42cf89d542e6b9b7ae71" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:24.911792 9798 raft_consensus.cc:2454] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:24.912894 6735 leader_election.cc:302] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:24.913285 10296 raft_consensus.cc:2781] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:24.913398 10296 raft_consensus.cc:482] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:24.913482 10296 raft_consensus.cc:3037] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.916846 9644 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 14:13:24.917418 10296 raft_consensus.cc:504] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.919240 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1949c165716e42cf89d542e6b9b7ae71" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:24.919451 9798 raft_consensus.cc:3037] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.924335 9798 raft_consensus.cc:2454] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:24.924858 6735 leader_election.cc:302] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:24.925181 10514 raft_consensus.cc:2781] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:24.925350 10514 raft_consensus.cc:686] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:24.925606 10514 consensus_queue.cc:237] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.926017 10296 leader_election.cc:288] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:24.929679 7060 raft_consensus.cc:1261] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.930364 10545 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.940898 10737 ts_tablet_manager.cc:1875] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:24.942399 10719 tablet_bootstrap.cc:492] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:24.942646 10719 ts_tablet_manager.cc:1367] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.290s user 0.010s sys 0.007s
I20240503 14:13:24.946368 10736 ts_tablet_manager.cc:1875] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:24.947252 10719 raft_consensus.cc:348] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:24.947618 10719 raft_consensus.cc:374] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:24.948009 9533 raft_consensus.cc:1261] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.948516 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1949c165716e42cf89d542e6b9b7ae71" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:24.948740 9531 raft_consensus.cc:2454] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:24.948746 10729 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.949012 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1949c165716e42cf89d542e6b9b7ae71" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:24.949189 9532 raft_consensus.cc:3037] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:24.949172 6865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 175434us
I20240503 14:13:24.949893 6658 catalog_manager.cc:5484] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:24.950709 6798 raft_consensus.cc:1261] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:24.951704 10731 ts_tablet_manager.cc:1875] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:24.955279 9532 raft_consensus.cc:2454] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:24.955806 10545 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:24.956249 10719 raft_consensus.cc:729] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:24.956960 6798 raft_consensus.cc:2932] T 20c4e7c22c904daebe97897320f6226e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:24.957722 10719 consensus_queue.cc:260] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:24.959743 10545 raft_consensus.cc:1050] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:24.960896 10545 consensus_queue.cc:237] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:24.962706 10719 ts_tablet_manager.cc:1398] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240503 14:13:24.965960 10250 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:24.966120 10250 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:24.966944 10250 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:24.967794 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:24.968092 7191 raft_consensus.cc:2454] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:24.972298 9600 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:24.972764 10250 raft_consensus.cc:2781] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:24.972884 10250 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:24.972991 10250 raft_consensus.cc:3037] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.975317 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:24.975734 10250 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
W20240503 14:13:24.976253 9600 leader_election.cc:341] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:24.976822 10250 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:24.977506 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:24.977754 7191 raft_consensus.cc:3037] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:24.979542 10736 ts_tablet_manager.cc:1888] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:24.979702 10736 log.cc:1196] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/0109f8f397f24bcabb6ebbf84a1139a3
I20240503 14:13:24.980155 10736 ts_tablet_manager.cc:1909] T 0109f8f397f24bcabb6ebbf84a1139a3 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:24.980784 7191 raft_consensus.cc:2454] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:24.981418 9600 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:24.981868 10250 raft_consensus.cc:2781] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:24.982422 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
W20240503 14:13:24.983172 9600 leader_election.cc:341] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:24.985396 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:24.986202 10731 ts_tablet_manager.cc:1888] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:24.986342 10731 log.cc:1196] T 20c4e7c22c904daebe97897320f6226e P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/20c4e7c22c904daebe97897320f6226e
I20240503 14:13:24.987125 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 20c4e7c22c904daebe97897320f6226e (table rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]) successfully deleted
I20240503 14:13:24.987826 10731 tablet_replica.cc:330] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:24.987864 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 0109f8f397f24bcabb6ebbf84a1139a3 (table rebalancer_dynamic_table_new_name [id=4558b2809da749ed90d80b6914f412e2]) successfully deleted
I20240503 14:13:24.988085 10731 raft_consensus.cc:2229] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:24.988483 10731 raft_consensus.cc:2258] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:24.989511 10731 ts_tablet_manager.cc:1875] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:24.999434 9779 tablet_service.cc:1467] Processing CreateTablet for tablet c06c4bfcd20947ae9e7bfd01e097e02a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]), partition=
I20240503 14:13:24.999912 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06c4bfcd20947ae9e7bfd01e097e02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.004004 6910 tablet_service.cc:1467] Processing CreateTablet for tablet c06c4bfcd20947ae9e7bfd01e097e02a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]), partition=
I20240503 14:13:25.004539 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06c4bfcd20947ae9e7bfd01e097e02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.005568 10740 raft_consensus.cc:482] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:25.005741 10740 raft_consensus.cc:504] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.007355 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7255e647ced644ed9b76937e7808a147" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:25.007644 9929 raft_consensus.cc:2454] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:25.008142 9467 leader_election.cc:302] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:25.008514 10637 raft_consensus.cc:2781] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:25.008648 10637 raft_consensus.cc:482] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:25.007582 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7255e647ced644ed9b76937e7808a147" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:25.008745 10637 raft_consensus.cc:3037] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.013204 10637 raft_consensus.cc:504] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.013182 10729 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:25.014770 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7255e647ced644ed9b76937e7808a147" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:25.014997 9929 raft_consensus.cc:3037] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.017506 10731 ts_tablet_manager.cc:1888] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:25.017671 10731 log.cc:1196] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/0109f8f397f24bcabb6ebbf84a1139a3
I20240503 14:13:25.017800 9929 raft_consensus.cc:2454] T 7255e647ced644ed9b76937e7808a147 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:25.018134 10731 ts_tablet_manager.cc:1909] T 0109f8f397f24bcabb6ebbf84a1139a3 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:25.018442 9467 leader_election.cc:302] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:25.018107 6799 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:25.019088 10637 leader_election.cc:288] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:25.020020 10718 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:25.020332 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 0109f8f397f24bcabb6ebbf84a1139a3 (table rebalancer_dynamic_table_new_name [id=4558b2809da749ed90d80b6914f412e2]) successfully deleted
I20240503 14:13:25.020376 10637 raft_consensus.cc:2781] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:25.020824 7060 raft_consensus.cc:1261] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:25.020874 10637 raft_consensus.cc:686] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:25.021471 7191 raft_consensus.cc:1261] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:25.021520 10545 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:25.022065 9276 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:25.023806 10718 ts_tablet_manager.cc:1367] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.284s user 0.008s sys 0.003s
I20240503 14:13:25.021468 10637 consensus_queue.cc:237] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.024420 7059 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:25.031575 10718 raft_consensus.cc:348] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.032490 10718 raft_consensus.cc:374] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.035861 10718 raft_consensus.cc:729] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.036871 10718 consensus_queue.cc:260] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.038287 6657 catalog_manager.cc:5484] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:25.042735 10718 ts_tablet_manager.cc:1398] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240503 14:13:25.048417 10718 tablet_bootstrap.cc:492] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:25.050441 10740 leader_election.cc:288] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:25.054100 10749 tablet_bootstrap.cc:492] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:25.057015 6654 catalog_manager.cc:5484] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:25.058614 10749 tablet_bootstrap.cc:654] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.062234 10749 tablet_bootstrap.cc:492] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:25.062472 10749 ts_tablet_manager.cc:1367] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:13:25.062186 10729 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.063452 7060 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.067916 6780 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240503 14:13:25.068917 7191 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.069854 10749 raft_consensus.cc:348] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.070096 10749 raft_consensus.cc:374] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.070175 10749 raft_consensus.cc:729] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.070829 10749 consensus_queue.cc:260] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.071645 10749 ts_tablet_manager.cc:1398] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 14:13:25.071862 6658 catalog_manager.cc:5484] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.073053 9663 raft_consensus.cc:2454] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:25.075229 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7255e647ced644ed9b76937e7808a147" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:25.076153 10718 tablet_bootstrap.cc:654] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:25.075764 9467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 57344.9us
I20240503 14:13:25.077977 9532 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:25.080119 9644 tablet_service.cc:1467] Processing CreateTablet for tablet c06c4bfcd20947ae9e7bfd01e097e02a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]), partition=
I20240503 14:13:25.080566 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06c4bfcd20947ae9e7bfd01e097e02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.083096 10718 tablet_bootstrap.cc:492] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:25.083293 10718 ts_tablet_manager.cc:1367] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240503 14:13:25.084237 10718 raft_consensus.cc:348] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.084501 10718 raft_consensus.cc:374] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.084630 10718 raft_consensus.cc:729] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.085239 9664 raft_consensus.cc:1261] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:25.085742 9663 raft_consensus.cc:3037] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.086900 10250 raft_consensus.cc:686] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:25.087262 10250 consensus_queue.cc:237] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.089001 10718 consensus_queue.cc:260] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.089658 10718 ts_tablet_manager.cc:1398] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:25.089895 6929 consensus_queue.cc:237] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:25.091303 9872 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:25.094595 6799 raft_consensus.cc:1261] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:25.095222 9663 raft_consensus.cc:2454] T 7255e647ced644ed9b76937e7808a147 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:25.095532 10729 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:25.101070 10715 tablet_bootstrap.cc:492] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:25.103541 10715 tablet_bootstrap.cc:654] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.104874 9872 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:25.106695 10737 ts_tablet_manager.cc:1888] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:25.106906 10737 log.cc:1196] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/0109f8f397f24bcabb6ebbf84a1139a3
I20240503 14:13:25.107921 10737 ts_tablet_manager.cc:1909] T 0109f8f397f24bcabb6ebbf84a1139a3 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:25.109223 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 0109f8f397f24bcabb6ebbf84a1139a3 (table rebalancer_dynamic_table_new_name [id=4558b2809da749ed90d80b6914f412e2]) successfully deleted
I20240503 14:13:25.109531 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9815af88a784ba7a50a43ca776ecd20 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:25.112097 6799 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:25.113102 6650 catalog_manager.cc:5484] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:25.119524 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet f9815af88a784ba7a50a43ca776ecd20 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:25.124511 9530 raft_consensus.cc:1261] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:25.125772 9872 consensus_queue.cc:1035] T f9815af88a784ba7a50a43ca776ecd20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:25.127436 9664 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.128612 9532 raft_consensus.cc:2932] T f9815af88a784ba7a50a43ca776ecd20 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:25.131834 10715 tablet_bootstrap.cc:492] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:25.132118 10715 ts_tablet_manager.cc:1367] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.004s
I20240503 14:13:25.132958 10715 raft_consensus.cc:348] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.133350 10715 raft_consensus.cc:374] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.133750 10715 raft_consensus.cc:729] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.133860 10756 tablet_replica.cc:330] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:25.134089 10756 raft_consensus.cc:2229] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:25.134125 10715 consensus_queue.cc:260] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.136567 10715 ts_tablet_manager.cc:1398] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:25.134488 10756 raft_consensus.cc:2258] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:25.139494 10756 ts_tablet_manager.cc:1875] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:25.150837 10756 ts_tablet_manager.cc:1888] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:25.152165 10756 log.cc:1196] T f9815af88a784ba7a50a43ca776ecd20 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9815af88a784ba7a50a43ca776ecd20
I20240503 14:13:25.153679 6650 catalog_manager.cc:5484] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:25.154163 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet f9815af88a784ba7a50a43ca776ecd20 (table rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]) successfully deleted
I20240503 14:13:25.234151 6929 consensus_queue.cc:237] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:25.237370 7191 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:25.238036 9276 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.240808 9872 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:25.241377 7191 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:25.247290 6650 catalog_manager.cc:5484] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:25.256098 6799 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:25.256762 9872 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:25.268520 6865 consensus_peers.cc:485] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 343520037eac42c4ac76fa1cc5e26fc3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:25.269814 10250 raft_consensus.cc:482] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:25.269995 10250 raft_consensus.cc:504] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.270752 10250 leader_election.cc:288] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:25.271245 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06c4bfcd20947ae9e7bfd01e097e02a" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:25.271478 6929 raft_consensus.cc:2454] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:25.271914 9599 leader_election.cc:302] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:25.272207 10250 raft_consensus.cc:2781] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:25.272310 10250 raft_consensus.cc:482] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:25.272226 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06c4bfcd20947ae9e7bfd01e097e02a" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:25.272411 10250 raft_consensus.cc:3037] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.272569 9798 raft_consensus.cc:2454] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:25.274986 10250 raft_consensus.cc:504] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.276518 10250 leader_election.cc:288] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:25.276839 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06c4bfcd20947ae9e7bfd01e097e02a" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:25.277053 6929 raft_consensus.cc:3037] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.279969 6929 raft_consensus.cc:2454] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:25.280404 9599 leader_election.cc:302] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:25.280733 10250 raft_consensus.cc:2781] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:25.280726 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06c4bfcd20947ae9e7bfd01e097e02a" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:25.280936 9798 raft_consensus.cc:3037] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.280985 10250 raft_consensus.cc:686] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:25.281234 10250 consensus_queue.cc:237] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.283883 9798 raft_consensus.cc:2454] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:25.287552 6799 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:25.296146 6650 catalog_manager.cc:5484] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:25.303241 10698 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:25.303454 10698 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.304369 10698 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:25.304306 9798 consensus_queue.cc:237] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:25.305145 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:25.305426 7191 raft_consensus.cc:2379] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3efdbd3506274dba846ae40404f0572c in current term 1: Already voted for candidate f03dc37fe3234033bc665bed4f1b51f5 in this term.
I20240503 14:13:25.305917 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:25.309567 9733 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:25.310096 10698 raft_consensus.cc:3037] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.313444 10698 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:25.315790 9929 raft_consensus.cc:1261] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:25.316017 9532 raft_consensus.cc:1261] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:25.316120 9733 consensus_peers.cc:485] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b556ba3bf82b40d988c0173f742a0ff2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:25.316452 10698 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:25.316738 10696 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:25.333312 10698 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:25.334926 9532 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:25.339439 9929 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:25.340684 6650 catalog_manager.cc:5484] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:25.359939 7060 consensus_queue.cc:237] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:25.362852 7191 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:25.363459 10371 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.366060 10750 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:25.369385 7191 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:25.373569 6929 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:25.374192 10785 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.377554 6650 catalog_manager.cc:5484] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
W20240503 14:13:25.381472 6995 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 669d722d3eb1475dba616ed72cb775ac. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:25.383931 6929 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:25.392473 7060 consensus_queue.cc:237] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:25.394677 6799 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:25.394894 7191 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:25.395089 6996 consensus_peers.cc:485] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3160f1a600604f258f76614f13429563. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:25.395223 10371 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.396062 10750 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.398370 10371 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:25.399066 6799 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:25.400502 7191 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:25.406042 6650 catalog_manager.cc:5484] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:25.411492 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:25.417917 6929 raft_consensus.cc:1261] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:25.418052 9798 raft_consensus.cc:1261] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:25.418700 10250 consensus_queue.cc:1035] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:25.418926 10744 consensus_queue.cc:1035] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:25.421258 6931 consensus_queue.cc:237] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:25.423902 6654 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]). Expected version 1 got 0
I20240503 14:13:25.426680 7191 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:25.427946 10729 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.432793 7060 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:25.433508 6865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e431272976034209b2c40cb0e460e17a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:25.435168 10514 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:25.437276 10729 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.439862 10808 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:25.439833 10729 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:25.443614 6658 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]). Expected version 1 got 0
I20240503 14:13:25.444934 7191 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:25.450103 10808 tablet.cc:1620] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:25.455020 6658 catalog_manager.cc:5576] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285) does not have the latest schema for tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]). Expected version 1 got 0
I20240503 14:13:25.478477 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b] alter complete (version 1)
I20240503 14:13:25.489033 10637 consensus_queue.cc:1035] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:25.507812 6654 catalog_manager.cc:5484] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:25.516220 10745 consensus_queue.cc:1035] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:25.566012 6734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 103913us
I20240503 14:13:25.570843 9798 consensus_queue.cc:237] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:25.579195 6658 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]). Expected version 1 got 0
I20240503 14:13:25.580374 9532 raft_consensus.cc:1261] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:25.581183 9664 raft_consensus.cc:1261] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:25.581264 10491 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:25.582686 10698 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:25.584573 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
W20240503 14:13:25.587225 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b18e1d58117421c82ba86c08cd1253b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:25.591744 10698 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:25.588172 10464 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20240503 14:13:25.596966 9530 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:25.609238 10250 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:25.613029 10812 log.cc:924] Time spent T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20240503 14:13:25.612831 6650 catalog_manager.cc:5484] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:25.613636 10807 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:25.616997 10807 tablet.cc:1620] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:25.619108 10814 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:25.619786 10814 tablet.cc:1620] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: MemRowSet was empty: no flush needed.
I20240503 14:13:25.619738 7060 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:25.621726 6651 catalog_manager.cc:5576] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285) does not have the latest schema for tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]). Expected version 2 got 0
I20240503 14:13:25.624441 9664 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:25.632254 6651 catalog_manager.cc:5576] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285) does not have the latest schema for tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]). Expected version 2 got 1
I20240503 14:13:25.648701 10836 ts_tablet_manager.cc:902] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:25.663921 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:25.679781 6658 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b]). Expected version 2 got 1
I20240503 14:13:25.682077 10814 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:25.682658 10814 tablet.cc:1620] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: MemRowSet was empty: no flush needed.
I20240503 14:13:25.682957 10808 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:25.683426 10808 tablet.cc:1620] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:25.690711 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 1461e0716aee4f74acefc639cf79eaf0 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.691195 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1461e0716aee4f74acefc639cf79eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.691294 10744 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:25.692620 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 1461e0716aee4f74acefc639cf79eaf0 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.693058 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1461e0716aee4f74acefc639cf79eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.694482 9643 tablet_service.cc:1467] Processing CreateTablet for tablet b6d600a8b0134a018f8c9e5a1d9b79c5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.695011 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d600a8b0134a018f8c9e5a1d9b79c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.694895 10807 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:25.695187 6910 tablet_service.cc:1467] Processing CreateTablet for tablet b6d600a8b0134a018f8c9e5a1d9b79c5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.695402 10807 tablet.cc:1620] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:25.695634 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d600a8b0134a018f8c9e5a1d9b79c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.700206 9512 tablet_service.cc:1467] Processing CreateTablet for tablet b6d600a8b0134a018f8c9e5a1d9b79c5 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.700321 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb6a131c94047649646ad83a0074133 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.700666 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d600a8b0134a018f8c9e5a1d9b79c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.700803 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb6a131c94047649646ad83a0074133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.704088 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb6a131c94047649646ad83a0074133 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.704684 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb6a131c94047649646ad83a0074133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.708313 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 1461e0716aee4f74acefc639cf79eaf0 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.708752 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1461e0716aee4f74acefc639cf79eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.714494 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb6a131c94047649646ad83a0074133 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:25.714841 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=17c396b039e54dceb4adf4a64676986b] alter complete (version 2)
I20240503 14:13:25.718446 6929 consensus_queue.cc:237] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:25.721555 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb6a131c94047649646ad83a0074133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.726125 9664 raft_consensus.cc:1261] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:25.728799 7190 consensus_queue.cc:237] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:25.729027 9872 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:25.733234 7059 raft_consensus.cc:1261] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:25.733875 9872 consensus_queue.cc:1035] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:25.740911 10729 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.744427 9664 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.749114 7059 raft_consensus.cc:2932] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.758225 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f5738d4265f941519b90b07334d06af9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:25.761735 6654 catalog_manager.cc:5484] T f5738d4265f941519b90b07334d06af9 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:25.767371 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet f5738d4265f941519b90b07334d06af9 with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:25.773293 10836 tablet_copy_client.cc:308] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:25.786026 10850 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:25.789543 10846 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:25.789902 10855 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:25.791826 10855 tablet_bootstrap.cc:654] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.792205 10849 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:25.795446 10849 tablet_bootstrap.cc:654] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.795764 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 343520037eac42c4ac76fa1cc5e26fc3 from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:39445)
I20240503 14:13:25.799078 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 343520037eac42c4ac76fa1cc5e26fc3 from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:39445: session id = ea801601713a4d55aee0de9e3473e0bc-343520037eac42c4ac76fa1cc5e26fc3
I20240503 14:13:25.799657 10846 tablet_bootstrap.cc:654] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.801002 10850 tablet_bootstrap.cc:654] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.801977 6951 tablet_copy_source_session.cc:205] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:25.803155 10845 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:25.805593 10836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 343520037eac42c4ac76fa1cc5e26fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.809873 10849 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:25.810232 10849 ts_tablet_manager.cc:1367] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240503 14:13:25.811283 10849 raft_consensus.cc:348] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.805765 10845 tablet_bootstrap.cc:654] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.813479 10849 raft_consensus.cc:374] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.813611 10849 raft_consensus.cc:729] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.814646 10849 consensus_queue.cc:260] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.818112 10846 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:25.818382 10846 ts_tablet_manager.cc:1367] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.003s
I20240503 14:13:25.819195 10849 ts_tablet_manager.cc:1398] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:25.819811 10851 tablet_replica.cc:330] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:25.820420 10836 tablet_copy_client.cc:783] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:25.820732 10851 raft_consensus.cc:2229] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:25.821094 10836 tablet_copy_client.cc:647] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:25.821089 7059 raft_consensus.cc:1261] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:25.821249 6929 raft_consensus.cc:1261] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:25.820467 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8014635e8294062b1c1985a100307ff. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:25.821847 10851 raft_consensus.cc:2258] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:25.821820 10201 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.822522 10574 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:25.822887 10849 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:25.824811 10849 tablet_bootstrap.cc:654] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.826452 10836 tablet_copy_client.cc:523] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:25.830147 10574 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:25.832939 10836 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:25.832615 6929 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:25.835094 7060 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:25.836213 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:25.836894 10850 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:25.837096 10850 ts_tablet_manager.cc:1367] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.007s
I20240503 14:13:25.838184 10846 raft_consensus.cc:348] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.838428 10846 raft_consensus.cc:374] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.838514 10846 raft_consensus.cc:729] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.837910 10850 raft_consensus.cc:348] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.838799 10850 raft_consensus.cc:374] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.838951 10850 raft_consensus.cc:729] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.839747 10846 consensus_queue.cc:260] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.839421 10850 consensus_queue.cc:260] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.841282 6658 catalog_manager.cc:5484] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:25.849179 10850 ts_tablet_manager.cc:1398] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:13:25.849882 10846 ts_tablet_manager.cc:1398] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20240503 14:13:25.860862 10807 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:25.861677 10814 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:25.870863 10868 ts_tablet_manager.cc:902] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:25.871553 10862 ts_tablet_manager.cc:902] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:25.875962 10808 tablet.cc:1722] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:25.881480 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=17c396b039e54dceb4adf4a64676986b] alter complete (version 3)
I20240503 14:13:25.881853 10849 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:25.882721 10871 ts_tablet_manager.cc:902] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:25.882881 10849 ts_tablet_manager.cc:1367] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.004s
I20240503 14:13:25.883870 10849 raft_consensus.cc:348] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.884896 10849 raft_consensus.cc:374] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.885146 10849 raft_consensus.cc:729] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
W20240503 14:13:25.885950 7125 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.2:45285, user_credentials={real_user=slave}} blocked reactor thread for 44187.4us
I20240503 14:13:25.886626 10849 consensus_queue.cc:260] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.888514 10845 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:25.888795 10845 ts_tablet_manager.cc:1367] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20240503 14:13:25.889757 10845 raft_consensus.cc:348] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.890420 10845 raft_consensus.cc:374] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.890579 10845 raft_consensus.cc:729] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.890714 10849 ts_tablet_manager.cc:1398] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:25.890966 10845 consensus_queue.cc:260] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:25.891844 10845 ts_tablet_manager.cc:1398] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:25.892156 10849 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:25.893771 10845 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:25.894063 10849 tablet_bootstrap.cc:654] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.895803 10845 tablet_bootstrap.cc:654] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.903705 10851 ts_tablet_manager.cc:1875] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:25.908555 10849 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:25.910046 10849 ts_tablet_manager.cc:1367] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20240503 14:13:25.910457 10845 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:25.910715 10845 ts_tablet_manager.cc:1367] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.009s
I20240503 14:13:25.911412 10849 raft_consensus.cc:348] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.911628 10845 raft_consensus.cc:348] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.912012 10845 raft_consensus.cc:374] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.912168 10845 raft_consensus.cc:729] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.912585 10845 consensus_queue.cc:260] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.912863 10871 tablet_copy_client.cc:308] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:25.914021 10849 raft_consensus.cc:374] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.914225 10849 raft_consensus.cc:729] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.914222 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet b556ba3bf82b40d988c0173f742a0ff2 from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:47571)
I20240503 14:13:25.914510 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet b556ba3bf82b40d988c0173f742a0ff2 from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:47571: session id = f1a669e28f014fe287932acf56bebc13-b556ba3bf82b40d988c0173f742a0ff2
I20240503 14:13:25.914630 10849 consensus_queue.cc:260] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.915450 10855 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:25.917191 10855 ts_tablet_manager.cc:1367] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.129s user 0.005s sys 0.014s
I20240503 14:13:25.919970 10849 ts_tablet_manager.cc:1398] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:25.921232 9819 tablet_copy_source_session.cc:205] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:25.922741 10845 ts_tablet_manager.cc:1398] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:13:25.922657 10855 raft_consensus.cc:348] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.923162 10845 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:25.923825 10871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b556ba3bf82b40d988c0173f742a0ff2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:25.924176 10855 raft_consensus.cc:374] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.924618 10855 raft_consensus.cc:729] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.925571 10845 tablet_bootstrap.cc:654] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:25.926513 10855 consensus_queue.cc:260] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.928387 10855 ts_tablet_manager.cc:1398] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:25.933737 10868 tablet_copy_client.cc:308] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:25.936204 10871 tablet_copy_client.cc:783] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:25.936652 10871 tablet_copy_client.cc:647] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:25.939431 10871 tablet_copy_client.cc:523] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:25.943449 10698 raft_consensus.cc:482] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:25.944018 10871 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:25.944697 10698 raft_consensus.cc:504] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.945614 10698 leader_election.cc:288] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:25.945663 10845 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:25.945827 10845 ts_tablet_manager.cc:1367] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.005s
I20240503 14:13:25.946789 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb6a131c94047649646ad83a0074133" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:25.946975 10845 raft_consensus.cc:348] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.947090 9532 raft_consensus.cc:2454] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:25.947185 10845 raft_consensus.cc:374] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:25.947275 10845 raft_consensus.cc:729] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:25.947494 10845 consensus_queue.cc:260] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:25.947942 9735 leader_election.cc:302] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:25.948256 10845 ts_tablet_manager.cc:1398] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:25.948355 10698 raft_consensus.cc:2781] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:25.948473 10698 raft_consensus.cc:482] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:25.948570 10698 raft_consensus.cc:3037] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.951535 10871 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:25.952112 10871 tablet_bootstrap.cc:492] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:25.952380 10871 ts_tablet_manager.cc:1367] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240503 14:13:25.953198 10871 raft_consensus.cc:348] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:25.953476 10871 raft_consensus.cc:729] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:25.951447 10698 raft_consensus.cc:504] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.953727 10871 consensus_queue.cc:260] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:25.954845 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb6a131c94047649646ad83a0074133" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:25.955070 9532 raft_consensus.cc:3037] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.955771 10871 ts_tablet_manager.cc:1398] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:25.956599 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-b556ba3bf82b40d988c0173f742a0ff2 received from {username='slave'} at 127.0.186.1:47571
I20240503 14:13:25.956746 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session f1a669e28f014fe287932acf56bebc13-b556ba3bf82b40d988c0173f742a0ff2 on tablet b556ba3bf82b40d988c0173f742a0ff2 with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:25.957907 9532 raft_consensus.cc:2454] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:25.958395 9735 leader_election.cc:302] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:25.958792 10491 raft_consensus.cc:2781] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:25.958962 10491 raft_consensus.cc:686] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:25.959302 10491 consensus_queue.cc:237] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.960633 10698 leader_election.cc:288] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:25.968854 6658 catalog_manager.cc:5484] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:25.975283 10744 raft_consensus.cc:482] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:25.975556 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb6a131c94047649646ad83a0074133" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:25.975543 10744 raft_consensus.cc:504] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:25.976111 9664 raft_consensus.cc:2454] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:25.976567 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb6a131c94047649646ad83a0074133" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:25.976594 10744 leader_election.cc:288] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:25.976763 9664 raft_consensus.cc:3037] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:25.977875 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb6a131c94047649646ad83a0074133" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:25.979687 9664 raft_consensus.cc:2454] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:25.981524 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb6a131c94047649646ad83a0074133" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:25.981827 9532 raft_consensus.cc:2379] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f03dc37fe3234033bc665bed4f1b51f5 in current term 1: Already voted for candidate 3efdbd3506274dba846ae40404f0572c in this term.
I20240503 14:13:25.983065 9601 leader_election.cc:302] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5; no voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:25.984386 6655 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:25.984659 6655 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:25.988859 10744 raft_consensus.cc:2726] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:25.991390 6655 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet c06c4bfcd20947ae9e7bfd01e097e02a
I20240503 14:13:25.993304 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet c06c4bfcd20947ae9e7bfd01e097e02a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:25 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:25.993646 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet c06c4bfcd20947ae9e7bfd01e097e02a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:25 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:25.994091 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet c06c4bfcd20947ae9e7bfd01e097e02a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:25 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:25.999665 10862 tablet_copy_client.cc:308] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:26.001003 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet e431272976034209b2c40cb0e460e17a from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:39445)
I20240503 14:13:26.001188 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet e431272976034209b2c40cb0e460e17a from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:39445: session id = ea801601713a4d55aee0de9e3473e0bc-e431272976034209b2c40cb0e460e17a
I20240503 14:13:26.004324 10836 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:26.005023 10836 tablet_bootstrap.cc:492] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:26.005578 6951 tablet_copy_source_session.cc:205] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:26.005865 10836 ts_tablet_manager.cc:1367] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.173s user 0.052s sys 0.004s
I20240503 14:13:26.009523 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e431272976034209b2c40cb0e460e17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.012214 10893 tablet_replica.cc:330] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:26.012585 10851 ts_tablet_manager.cc:1888] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:26.012603 10893 raft_consensus.cc:2229] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:26.013124 10893 raft_consensus.cc:2258] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:26.009559 10836 raft_consensus.cc:348] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:26.013803 10836 raft_consensus.cc:729] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:26.014060 10836 consensus_queue.cc:260] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:26.014518 10893 ts_tablet_manager.cc:1875] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:26.014901 10836 ts_tablet_manager.cc:1398] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:13:26.015918 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-343520037eac42c4ac76fa1cc5e26fc3 received from {username='slave'} at 127.0.186.8:39445
I20240503 14:13:26.016079 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-343520037eac42c4ac76fa1cc5e26fc3 on tablet 343520037eac42c4ac76fa1cc5e26fc3 with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:26.022766 10893 ts_tablet_manager.cc:1888] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:26.023012 10893 log.cc:1196] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/c06c4bfcd20947ae9e7bfd01e097e02a
I20240503 14:13:26.023380 10851 log.cc:1196] T f5738d4265f941519b90b07334d06af9 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/f5738d4265f941519b90b07334d06af9
I20240503 14:13:26.023772 10893 ts_tablet_manager.cc:1909] T c06c4bfcd20947ae9e7bfd01e097e02a P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:26.024551 10892 tablet_replica.cc:330] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:26.024786 10892 raft_consensus.cc:2229] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:26.025051 10892 raft_consensus.cc:2258] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:26.026222 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table_new_name [id=17c396b039e54dceb4adf4a64676986b]) successfully deleted
I20240503 14:13:26.028270 10871 ts_tablet_manager.cc:902] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:26.030938 10848 raft_consensus.cc:482] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:26.031111 10848 raft_consensus.cc:504] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:26.031946 10848 leader_election.cc:288] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:26.032218 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet f5738d4265f941519b90b07334d06af9 (table rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]) successfully deleted
I20240503 14:13:26.033211 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d600a8b0134a018f8c9e5a1d9b79c5" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:26.033583 9532 raft_consensus.cc:2454] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:26.034108 6867 leader_election.cc:302] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:26.034396 10848 raft_consensus.cc:2781] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:26.034529 10848 raft_consensus.cc:482] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:26.034634 10848 raft_consensus.cc:3037] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.035940 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d600a8b0134a018f8c9e5a1d9b79c5" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:26.036195 9664 raft_consensus.cc:2454] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:26.038044 10848 raft_consensus.cc:504] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:26.038306 10894 tablet_replica.cc:330] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:26.038975 10848 leader_election.cc:288] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:26.039321 10894 raft_consensus.cc:2229] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:26.039763 10894 raft_consensus.cc:2258] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:26.039788 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d600a8b0134a018f8c9e5a1d9b79c5" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:26.040004 9530 raft_consensus.cc:3037] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.042654 9530 raft_consensus.cc:2454] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:26.044351 6867 leader_election.cc:302] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:26.044683 10729 raft_consensus.cc:2781] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:26.044827 10729 raft_consensus.cc:686] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:26.045114 10729 consensus_queue.cc:237] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:26.046053 9930 raft_consensus.cc:1203] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:26.047816 10894 ts_tablet_manager.cc:1875] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:26.048370 10862 tablet_copy_client.cc:783] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:26.048725 10862 tablet_copy_client.cc:647] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:26.050786 10862 tablet_copy_client.cc:523] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:26.053099 6930 consensus_queue.cc:237] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:26.055065 6655 catalog_manager.cc:5484] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:26.057889 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d600a8b0134a018f8c9e5a1d9b79c5" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:26.058133 9664 raft_consensus.cc:3037] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.059841 10862 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:26.060966 9664 raft_consensus.cc:2454] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:26.062484 10744 raft_consensus.cc:482] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:26.062631 10744 raft_consensus.cc:504] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:26.063320 10744 leader_election.cc:288] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:26.064157 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461e0716aee4f74acefc639cf79eaf0" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:26.064424 9930 raft_consensus.cc:2454] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:26.065016 9599 leader_election.cc:302] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:26.065466 10744 raft_consensus.cc:2781] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:26.065577 10744 raft_consensus.cc:482] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:26.065660 10744 raft_consensus.cc:3037] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.073256 7060 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:26.073467 7190 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:26.073724 6865 consensus_peers.cc:485] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34f1ed0cfa8948048b3762919481367c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:26.073932 10848 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:26.074949 10848 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:26.075373 10744 raft_consensus.cc:504] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:26.076838 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461e0716aee4f74acefc639cf79eaf0" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:26.077117 9930 raft_consensus.cc:3037] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.080575 10744 leader_election.cc:288] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:26.084821 9930 raft_consensus.cc:2454] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:26.085772 9599 leader_election.cc:302] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:26.086318 10894 ts_tablet_manager.cc:1888] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:26.086337 10828 raft_consensus.cc:2781] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:26.086498 10828 raft_consensus.cc:686] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:26.086802 10828 consensus_queue.cc:237] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:26.095337 10894 log.cc:1196] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/c06c4bfcd20947ae9e7bfd01e097e02a
I20240503 14:13:26.096081 10894 ts_tablet_manager.cc:1909] T c06c4bfcd20947ae9e7bfd01e097e02a P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:26.096256 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:26.096911 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 669d722d3eb1475dba616ed72cb775ac from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:48891)
I20240503 14:13:26.097118 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 669d722d3eb1475dba616ed72cb775ac from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:48891: session id = f03dc37fe3234033bc665bed4f1b51f5-669d722d3eb1475dba616ed72cb775ac
I20240503 14:13:26.100652 7081 tablet_copy_source_session.cc:205] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:26.102403 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461e0716aee4f74acefc639cf79eaf0" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:26.102783 9530 raft_consensus.cc:3037] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.102735 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table_new_name [id=17c396b039e54dceb4adf4a64676986b]) successfully deleted
I20240503 14:13:26.106065 9530 raft_consensus.cc:2454] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:26.106294 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461e0716aee4f74acefc639cf79eaf0" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:26.106827 9532 raft_consensus.cc:2362] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1. Re-sending same reply.
I20240503 14:13:26.107517 10868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669d722d3eb1475dba616ed72cb775ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.111994 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 48831ea899c445e199c5e455b5ef5b88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888]), partition=
I20240503 14:13:26.112435 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48831ea899c445e199c5e455b5ef5b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.114982 10868 tablet_copy_client.cc:783] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:26.115226 10868 tablet_copy_client.cc:647] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:26.116851 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 48831ea899c445e199c5e455b5ef5b88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888]), partition=
I20240503 14:13:26.117386 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48831ea899c445e199c5e455b5ef5b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.122545 10868 tablet_copy_client.cc:523] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:26.133970 10846 tablet_bootstrap.cc:492] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:26.136556 10846 tablet_bootstrap.cc:654] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.150663 10848 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:26.154444 10892 ts_tablet_manager.cc:1875] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:26.163260 10868 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:26.167955 6658 catalog_manager.cc:5484] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:26.170962 9600 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.4:44337, user_credentials={real_user=slave}} blocked reactor thread for 83192.3us
I20240503 14:13:26.173640 10846 tablet_bootstrap.cc:492] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:26.174163 10846 ts_tablet_manager.cc:1367] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.040s user 0.012s sys 0.000s
I20240503 14:13:26.176376 10846 raft_consensus.cc:348] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.176729 10846 raft_consensus.cc:374] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.176849 10846 raft_consensus.cc:729] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.177211 10846 consensus_queue.cc:260] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.182480 10908 tablet_bootstrap.cc:492] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:26.184298 10908 tablet_bootstrap.cc:654] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.187608 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 48831ea899c445e199c5e455b5ef5b88 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888]), partition=
I20240503 14:13:26.187644 6658 catalog_manager.cc:5484] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:26.187999 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48831ea899c445e199c5e455b5ef5b88. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.193594 7060 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:26.196359 10845 tablet_bootstrap.cc:492] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:26.198127 10845 tablet_bootstrap.cc:654] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.224562 10846 ts_tablet_manager.cc:1398] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240503 14:13:26.221958 7190 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:26.229660 10845 tablet_bootstrap.cc:492] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:26.229930 10845 ts_tablet_manager.cc:1367] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.007s
I20240503 14:13:26.230804 10845 raft_consensus.cc:348] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.231227 10845 raft_consensus.cc:374] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.231356 10845 raft_consensus.cc:729] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.235303 10871 tablet_copy_client.cc:308] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:26.236240 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 0b18e1d58117421c82ba86c08cd1253b from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:47571)
I20240503 14:13:26.236449 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 0b18e1d58117421c82ba86c08cd1253b from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:47571: session id = f1a669e28f014fe287932acf56bebc13-0b18e1d58117421c82ba86c08cd1253b
I20240503 14:13:26.240543 9819 tablet_copy_source_session.cc:205] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:26.251350 10895 ts_tablet_manager.cc:902] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:26.252763 10871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b18e1d58117421c82ba86c08cd1253b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.253732 10908 tablet_bootstrap.cc:492] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:26.253952 10908 ts_tablet_manager.cc:1367] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20240503 14:13:26.254784 10908 raft_consensus.cc:348] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.255076 10908 raft_consensus.cc:374] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.255230 10908 raft_consensus.cc:729] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.255573 10908 consensus_queue.cc:260] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.264497 10908 ts_tablet_manager.cc:1398] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:13:26.265291 10892 ts_tablet_manager.cc:1888] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:26.265414 10892 log.cc:1196] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/c06c4bfcd20947ae9e7bfd01e097e02a
I20240503 14:13:26.265831 10892 ts_tablet_manager.cc:1909] T c06c4bfcd20947ae9e7bfd01e097e02a P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:26.266820 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet c06c4bfcd20947ae9e7bfd01e097e02a (table rebalancer_dynamic_table_new_name [id=17c396b039e54dceb4adf4a64676986b]) successfully deleted
I20240503 14:13:26.269162 10871 tablet_copy_client.cc:783] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:26.269508 10871 tablet_copy_client.cc:647] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:26.272080 10871 tablet_copy_client.cc:523] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:26.275668 10871 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:26.277033 9930 consensus_queue.cc:237] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:26.288659 10845 consensus_queue.cc:260] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.293920 10845 ts_tablet_manager.cc:1398] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.064s user 0.002s sys 0.000s
W20240503 14:13:26.298830 9863 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4d6c0c0b6c249208e67400face0b9a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:26.300490 9797 raft_consensus.cc:1261] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:26.300835 10491 raft_consensus.cc:482] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:26.300998 7059 raft_consensus.cc:1261] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:26.301010 10491 raft_consensus.cc:504] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.301551 10111 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:26.301739 10491 leader_election.cc:288] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:26.302595 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48831ea899c445e199c5e455b5ef5b88" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:26.302821 6799 raft_consensus.cc:2454] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:26.303961 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48831ea899c445e199c5e455b5ef5b88" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:26.304230 9664 raft_consensus.cc:2454] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:26.303524 9733 leader_election.cc:302] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:26.305752 10698 raft_consensus.cc:2781] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:26.305857 10698 raft_consensus.cc:482] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:26.305961 10698 raft_consensus.cc:3037] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.309815 10111 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:26.310917 10698 raft_consensus.cc:504] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.312749 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48831ea899c445e199c5e455b5ef5b88" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:26.312975 9663 raft_consensus.cc:3037] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.313143 10698 leader_election.cc:288] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:26.319159 6798 raft_consensus.cc:1203] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:26.323554 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48831ea899c445e199c5e455b5ef5b88" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:26.323887 6799 raft_consensus.cc:3037] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.327222 6799 raft_consensus.cc:2454] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:26.328258 9733 leader_election.cc:302] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:26.328611 10696 raft_consensus.cc:2781] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:26.328770 10696 raft_consensus.cc:686] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:26.329108 10696 consensus_queue.cc:237] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.334867 9663 raft_consensus.cc:2454] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:26.337749 10871 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:26.338234 10871 tablet_bootstrap.cc:492] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:26.338529 10871 ts_tablet_manager.cc:1367] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.005s
I20240503 14:13:26.339398 10871 raft_consensus.cc:348] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:26.339675 10871 raft_consensus.cc:729] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:26.339892 10871 consensus_queue.cc:260] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:26.340544 10871 ts_tablet_manager.cc:1398] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20240503 14:13:26.341362 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-0b18e1d58117421c82ba86c08cd1253b received from {username='slave'} at 127.0.186.1:47571
I20240503 14:13:26.341501 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session f1a669e28f014fe287932acf56bebc13-0b18e1d58117421c82ba86c08cd1253b on tablet 0b18e1d58117421c82ba86c08cd1253b with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:26.363345 6655 catalog_manager.cc:5484] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:26.382591 10696 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.391949 10862 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:26.392580 10862 tablet_bootstrap.cc:492] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:26.392964 10862 ts_tablet_manager.cc:1367] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.333s user 0.060s sys 0.039s
I20240503 14:13:26.394101 10862 raft_consensus.cc:348] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:26.394488 10862 raft_consensus.cc:729] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:26.394759 10862 consensus_queue.cc:260] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:26.395817 10862 ts_tablet_manager.cc:1398] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240503 14:13:26.397925 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-e431272976034209b2c40cb0e460e17a received from {username='slave'} at 127.0.186.8:39445
I20240503 14:13:26.398061 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-e431272976034209b2c40cb0e460e17a on tablet e431272976034209b2c40cb0e460e17a with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:26.398646 10929 ts_tablet_manager.cc:902] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:26.401329 10868 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:26.402199 10868 tablet_bootstrap.cc:492] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:26.402534 10868 ts_tablet_manager.cc:1367] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.239s user 0.053s sys 0.006s
I20240503 14:13:26.403481 10868 raft_consensus.cc:348] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:26.403820 10868 raft_consensus.cc:729] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:26.404608 10868 consensus_queue.cc:260] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:26.405319 10868 ts_tablet_manager.cc:1398] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:26.406203 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-669d722d3eb1475dba616ed72cb775ac received from {username='slave'} at 127.0.186.6:48891
I20240503 14:13:26.406329 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-669d722d3eb1475dba616ed72cb775ac on tablet 669d722d3eb1475dba616ed72cb775ac with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:26.415772 10933 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:26.417356 7059 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:26.417706 9797 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:26.419247 10895 tablet_copy_client.cc:308] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:26.420006 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 3160f1a600604f258f76614f13429563 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:58913)
I20240503 14:13:26.420167 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 3160f1a600604f258f76614f13429563 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:58913: session id = 3efdbd3506274dba846ae40404f0572c-3160f1a600604f258f76614f13429563
I20240503 14:13:26.423352 7081 tablet_copy_source_session.cc:205] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:26.425477 10491 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.428350 6655 catalog_manager.cc:5484] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:26.431706 10895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3160f1a600604f258f76614f13429563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.449506 10895 tablet_copy_client.cc:783] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:26.450191 10895 tablet_copy_client.cc:647] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:26.454476 10929 tablet_copy_client.cc:308] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:26.455183 10895 tablet_copy_client.cc:523] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:26.455161 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet b8014635e8294062b1c1985a100307ff from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:38461)
I20240503 14:13:26.455325 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet b8014635e8294062b1c1985a100307ff from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:38461: session id = f03dc37fe3234033bc665bed4f1b51f5-b8014635e8294062b1c1985a100307ff
I20240503 14:13:26.457427 7211 tablet_copy_source_session.cc:205] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:26.458724 10895 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:26.458886 10929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8014635e8294062b1c1985a100307ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.465322 10848 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.467170 6798 raft_consensus.cc:1203] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:26.478083 10929 tablet_copy_client.cc:783] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:26.478515 10929 tablet_copy_client.cc:647] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:26.479795 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:26.480875 10929 tablet_copy_client.cc:523] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:26.489650 10929 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:26.494674 6799 raft_consensus.cc:1261] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:26.496245 10698 consensus_queue.cc:1035] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.501478 9663 raft_consensus.cc:1261] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:26.502192 10698 consensus_queue.cc:1035] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.503710 10729 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.506127 10729 raft_consensus.cc:1050] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:26.507220 10729 consensus_queue.cc:237] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:26.520081 7190 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:26.520830 9872 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:26.523386 9930 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:26.524351 9276 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:26.525651 6798 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:26.527712 9276 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:26.535359 6655 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 48831ea899c445e199c5e455b5ef5b88 (table rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888]). Expected version 1 got 0
I20240503 14:13:26.551628 10895 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:26.552357 10895 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:26.552762 10895 ts_tablet_manager.cc:1367] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.094s user 0.058s sys 0.032s
I20240503 14:13:26.553689 10895 raft_consensus.cc:348] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:26.554139 10895 raft_consensus.cc:729] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:26.554471 10895 consensus_queue.cc:260] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:26.556829 10895 ts_tablet_manager.cc:1398] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:26.557622 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-3160f1a600604f258f76614f13429563 received from {username='slave'} at 127.0.186.7:58913
I20240503 14:13:26.557669 10957 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:26.557804 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-3160f1a600604f258f76614f13429563 on tablet 3160f1a600604f258f76614f13429563 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:26.559172 10958 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:26.560424 9872 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:26.561209 10958 tablet.cc:1620] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:26.561554 6798 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:26.566586 6658 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 48831ea899c445e199c5e455b5ef5b88 (table rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888]). Expected version 1 got 0
I20240503 14:13:26.567853 10957 tablet.cc:1620] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:26.568768 9930 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:26.569622 6655 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 48831ea899c445e199c5e455b5ef5b88 (table rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888]). Expected version 1 got 0
I20240503 14:13:26.570626 6658 catalog_manager.cc:5484] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:26.573103 7191 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:26.585337 6651 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 48831ea899c445e199c5e455b5ef5b88 (table rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888]). Expected version 1 got 0
I20240503 14:13:26.594911 6651 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 48831ea899c445e199c5e455b5ef5b88 (table rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888]). Expected version 1 got 0
I20240503 14:13:26.598886 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888] alter complete (version 1)
I20240503 14:13:26.605372 10963 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:26.606040 10963 tablet.cc:1620] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:26.620477 9930 raft_consensus.cc:1203] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:26.631953 9797 raft_consensus.cc:1203] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:26.632109 10929 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:26.632652 10929 tablet_bootstrap.cc:492] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:26.632972 10929 ts_tablet_manager.cc:1367] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.143s user 0.049s sys 0.000s
I20240503 14:13:26.633777 10929 raft_consensus.cc:348] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:26.634038 10929 raft_consensus.cc:729] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:26.634771 10929 consensus_queue.cc:260] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:26.636315 10929 ts_tablet_manager.cc:1398] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:26.637017 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-b8014635e8294062b1c1985a100307ff received from {username='slave'} at 127.0.186.6:38461
I20240503 14:13:26.637166 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-b8014635e8294062b1c1985a100307ff on tablet b8014635e8294062b1c1985a100307ff with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:26.648962 10250 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.676283 10862 ts_tablet_manager.cc:902] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:26.677513 10905 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:26.705003 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:26.717835 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:26.720903 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca50346c04c427eb59e9f036a27dccc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.721366 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca50346c04c427eb59e9f036a27dccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.722108 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca50346c04c427eb59e9f036a27dccc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.722525 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca50346c04c427eb59e9f036a27dccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.724691 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 8477be3143f64d83a4c68f260a66efbf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.725121 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8477be3143f64d83a4c68f260a66efbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.726562 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 8477be3143f64d83a4c68f260a66efbf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.726935 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8477be3143f64d83a4c68f260a66efbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.728478 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 458ebc1295ee49999586eb699a29bbf8 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.728935 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458ebc1295ee49999586eb699a29bbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.728480 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 8477be3143f64d83a4c68f260a66efbf (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.734907 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8477be3143f64d83a4c68f260a66efbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.734913 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca50346c04c427eb59e9f036a27dccc (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.735493 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca50346c04c427eb59e9f036a27dccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.743994 10957 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:26.744136 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 458ebc1295ee49999586eb699a29bbf8 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.744513 10957 tablet.cc:1620] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:26.744757 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458ebc1295ee49999586eb699a29bbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.751103 10908 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:26.752316 10963 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:26.752856 10963 tablet.cc:1620] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:26.753767 10908 tablet_bootstrap.cc:654] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.762382 10958 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:26.762952 10958 tablet.cc:1620] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:26.766434 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 458ebc1295ee49999586eb699a29bbf8 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:26.766976 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458ebc1295ee49999586eb699a29bbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.775871 10845 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:26.778007 10845 tablet_bootstrap.cc:654] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.780779 10845 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:26.781049 10845 ts_tablet_manager.cc:1367] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:26.781965 10845 raft_consensus.cc:348] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:26.782306 10845 raft_consensus.cc:374] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.782474 10845 raft_consensus.cc:729] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.782866 10845 consensus_queue.cc:260] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:26.784967 10845 ts_tablet_manager.cc:1398] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:26.794417 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=3eda6fb4810644cfa87f453aa2b76888] alter complete (version 2)
I20240503 14:13:26.795974 10989 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:26.798305 10862 tablet_copy_client.cc:308] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:26.799135 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 34f1ed0cfa8948048b3762919481367c from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:39445)
I20240503 14:13:26.799328 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 34f1ed0cfa8948048b3762919481367c from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:39445: session id = ea801601713a4d55aee0de9e3473e0bc-34f1ed0cfa8948048b3762919481367c
I20240503 14:13:26.800427 10989 tablet_bootstrap.cc:654] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.802418 6951 tablet_copy_source_session.cc:205] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:26.807555 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f1ed0cfa8948048b3762919481367c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:13:26.812036 6734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 54996.9us
W20240503 14:13:26.812919 7126 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 46845.1us
I20240503 14:13:26.814220 10991 tablet_bootstrap.cc:492] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:26.814468 10983 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:26.814646 10986 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:26.815022 10908 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:26.815819 10908 ts_tablet_manager.cc:1367] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.007s
I20240503 14:13:26.816740 10987 tablet_bootstrap.cc:492] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:26.818282 10983 tablet_bootstrap.cc:654] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.819357 10908 raft_consensus.cc:348] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.819785 10908 raft_consensus.cc:374] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.819945 10908 raft_consensus.cc:729] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.821642 10908 consensus_queue.cc:260] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.827543 10987 tablet_bootstrap.cc:654] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.827993 10991 tablet_bootstrap.cc:654] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.828075 10908 ts_tablet_manager.cc:1398] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240503 14:13:26.828270 10986 tablet_bootstrap.cc:654] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.831800 10908 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:26.834069 10986 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:26.835882 10991 tablet_bootstrap.cc:492] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:26.836884 10908 tablet_bootstrap.cc:654] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.837549 10991 ts_tablet_manager.cc:1367] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.003s
I20240503 14:13:26.838603 10991 raft_consensus.cc:348] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.838904 10991 raft_consensus.cc:374] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.838971 10983 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:26.839017 10991 raft_consensus.cc:729] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.839123 10983 ts_tablet_manager.cc:1367] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240503 14:13:26.839670 10986 ts_tablet_manager.cc:1367] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240503 14:13:26.839769 10991 consensus_queue.cc:260] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.840611 10986 raft_consensus.cc:348] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.840886 10986 raft_consensus.cc:374] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.841028 10986 raft_consensus.cc:729] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.841315 10986 consensus_queue.cc:260] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.841413 10983 raft_consensus.cc:348] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.842278 10983 raft_consensus.cc:374] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.842442 10983 raft_consensus.cc:729] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.843312 10989 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:26.843482 10989 ts_tablet_manager.cc:1367] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.011s
I20240503 14:13:26.844444 10989 raft_consensus.cc:348] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:26.844673 10989 raft_consensus.cc:374] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.844787 10989 raft_consensus.cc:729] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.845072 10989 consensus_queue.cc:260] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:26.845739 10989 ts_tablet_manager.cc:1398] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:26.846957 10983 consensus_queue.cc:260] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.847710 10983 ts_tablet_manager.cc:1398] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:13:26.849038 10986 ts_tablet_manager.cc:1398] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:26.849409 10986 tablet_bootstrap.cc:492] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:26.849720 10862 tablet_copy_client.cc:783] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:26.850025 10862 tablet_copy_client.cc:647] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:26.853332 10862 tablet_copy_client.cc:523] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:26.855460 10991 ts_tablet_manager.cc:1398] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240503 14:13:26.857021 10862 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:26.860954 9664 raft_consensus.cc:1203] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:26.862776 10908 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:26.863020 10908 ts_tablet_manager.cc:1367] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.007s
I20240503 14:13:26.863315 10986 tablet_bootstrap.cc:654] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:26.863813 10987 tablet_bootstrap.cc:492] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:26.863965 10908 raft_consensus.cc:348] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:26.864030 10987 ts_tablet_manager.cc:1367] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240503 14:13:26.864259 10908 raft_consensus.cc:374] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.864352 10908 raft_consensus.cc:729] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.864699 10908 consensus_queue.cc:260] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:26.868533 10908 ts_tablet_manager.cc:1398] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:26.868753 10987 raft_consensus.cc:348] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.870223 10987 raft_consensus.cc:374] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.870380 10987 raft_consensus.cc:729] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.871220 10987 consensus_queue.cc:260] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.873399 10987 ts_tablet_manager.cc:1398] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:26.882351 10986 tablet_bootstrap.cc:492] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:26.882642 10986 ts_tablet_manager.cc:1367] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240503 14:13:26.883529 10986 raft_consensus.cc:348] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.883904 10986 raft_consensus.cc:374] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:26.884049 10986 raft_consensus.cc:729] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:26.884568 10986 consensus_queue.cc:260] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:26.885378 10986 ts_tablet_manager.cc:1398] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:26.900431 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:26.905169 10464 raft_consensus.cc:482] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:26.905380 10464 raft_consensus.cc:504] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.906054 10464 leader_election.cc:288] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:26.906767 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8477be3143f64d83a4c68f260a66efbf" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:26.906991 9797 raft_consensus.cc:2454] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:26.907500 6735 leader_election.cc:302] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:26.908303 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8477be3143f64d83a4c68f260a66efbf" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:26.908584 6931 raft_consensus.cc:2454] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:26.909339 10464 raft_consensus.cc:2781] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:26.909438 10464 raft_consensus.cc:482] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:26.909559 10464 raft_consensus.cc:3037] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.915374 10464 raft_consensus.cc:504] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.916029 10957 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:26.916782 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8477be3143f64d83a4c68f260a66efbf" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:26.916977 9797 raft_consensus.cc:3037] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.917390 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8477be3143f64d83a4c68f260a66efbf" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:26.917660 6931 raft_consensus.cc:3037] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:26.919343 10963 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:26.919799 9797 raft_consensus.cc:2454] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:26.920217 6735 leader_election.cc:302] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:26.923398 10985 raft_consensus.cc:2781] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:26.924274 6931 raft_consensus.cc:2454] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:26.925225 10464 leader_election.cc:288] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:26.928467 9664 raft_consensus.cc:1203] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:26.942497 11006 ts_tablet_manager.cc:902] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:26.944976 10985 raft_consensus.cc:686] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:26.945434 10985 consensus_queue.cc:237] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:26.947136 10698 raft_consensus.cc:1050] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:26.948485 10698 consensus_queue.cc:237] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:26.949417 11013 raft_consensus.cc:1050] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:26.953234 11013 consensus_queue.cc:237] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:26.956516 11006 tablet_copy_client.cc:308] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:26.957720 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=3eda6fb4810644cfa87f453aa2b76888] alter complete (version 3)
I20240503 14:13:26.957755 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet d4d6c0c0b6c249208e67400face0b9a6 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:44207)
I20240503 14:13:26.958025 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet d4d6c0c0b6c249208e67400face0b9a6 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:44207: session id = 7186ed8510424fd597173eaf10c57b12-d4d6c0c0b6c249208e67400face0b9a6
I20240503 14:13:26.960060 6651 catalog_manager.cc:5484] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:26.961972 9950 tablet_copy_source_session.cc:205] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:26.964828 11006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d6c0c0b6c249208e67400face0b9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:26.973649 9530 raft_consensus.cc:1261] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:26.974547 10696 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:26.975471 9530 raft_consensus.cc:1261] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:26.976604 10696 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:13:26.979746 9733 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.1:34101, user_credentials={real_user=slave}} blocked reactor thread for 48858.3us
I20240503 14:13:26.980947 9930 raft_consensus.cc:1261] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:26.981760 10958 tablet.cc:1722] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:26.982096 9665 raft_consensus.cc:1261] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:26.984834 6799 raft_consensus.cc:1261] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:26.986023 6798 raft_consensus.cc:1261] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:26.986860 11012 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:26.987267 11009 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:26.987514 11008 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:26.988337 11006 tablet_copy_client.cc:783] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:26.988691 11006 tablet_copy_client.cc:647] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:26.994165 11012 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:26.996109 11008 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:26.998356 9929 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.004475 6651 catalog_manager.cc:5484] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.004848 9532 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.008038 11006 tablet_copy_client.cc:523] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:27.011730 6798 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.014822 10698 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.020359 10750 raft_consensus.cc:1050] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:27.021533 10750 consensus_queue.cc:237] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:27.021611 9532 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.022171 9664 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.028963 6655 catalog_manager.cc:5484] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.029070 9798 consensus_queue.cc:237] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:27.031800 6799 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.035336 9798 consensus_queue.cc:237] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:27.041103 6800 raft_consensus.cc:1261] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:27.041355 6799 raft_consensus.cc:1261] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:27.042769 9664 raft_consensus.cc:1261] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:27.043514 9929 raft_consensus.cc:1261] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:27.043700 11009 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:27.044186 11009 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:27.045097 11009 consensus_queue.cc:1035] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:27.047308 11006 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:27.048935 11009 consensus_queue.cc:1035] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:27.049790 10698 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.055058 7191 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.056210 9798 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.056218 10371 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.057091 10371 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.059119 6798 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.059968 10785 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.062588 11009 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.066846 10862 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:27.067392 10862 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:27.067798 10862 ts_tablet_manager.cc:1367] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.211s user 0.049s sys 0.006s
I20240503 14:13:27.068773 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b18e1d58117421c82ba86c08cd1253b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:27.070161 10862 raft_consensus.cc:348] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:27.070538 10862 raft_consensus.cc:729] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:27.070816 10862 consensus_queue.cc:260] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:27.071575 10862 ts_tablet_manager.cc:1398] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:27.072360 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-34f1ed0cfa8948048b3762919481367c received from {username='slave'} at 127.0.186.8:39445
I20240503 14:13:27.072518 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-34f1ed0cfa8948048b3762919481367c on tablet 34f1ed0cfa8948048b3762919481367c with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:27.072278 6798 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.074250 6655 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:27.074476 6655 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:27.079437 6655 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 48831ea899c445e199c5e455b5ef5b88
I20240503 14:13:27.081158 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 48831ea899c445e199c5e455b5ef5b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:27.082067 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 48831ea899c445e199c5e455b5ef5b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:27.082312 9665 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.084232 10750 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:27.088480 7190 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:27.089299 9929 raft_consensus.cc:2932] T b556ba3bf82b40d988c0173f742a0ff2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.091729 6800 raft_consensus.cc:2932] T 0b18e1d58117421c82ba86c08cd1253b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:27.097959 6651 catalog_manager.cc:5484] T b556ba3bf82b40d988c0173f742a0ff2 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:27.099319 11039 tablet_replica.cc:330] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:27.099574 11039 raft_consensus.cc:2229] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:27.100000 11039 raft_consensus.cc:2258] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.101053 11039 ts_tablet_manager.cc:1875] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.103524 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet b556ba3bf82b40d988c0173f742a0ff2 with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:27.105755 6651 catalog_manager.cc:5484] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:27.109592 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b556ba3bf82b40d988c0173f742a0ff2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:27.110882 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b18e1d58117421c82ba86c08cd1253b with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:27.112638 11039 ts_tablet_manager.cc:1888] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:27.113282 11039 log.cc:1196] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/48831ea899c445e199c5e455b5ef5b88
I20240503 14:13:27.113970 11039 ts_tablet_manager.cc:1909] T 48831ea899c445e199c5e455b5ef5b88 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:27.112418 9798 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:27.115438 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 48831ea899c445e199c5e455b5ef5b88 (table rebalancer_dynamic_table_new_name [id=3eda6fb4810644cfa87f453aa2b76888]) successfully deleted
I20240503 14:13:27.122225 6798 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:27.126315 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 48831ea899c445e199c5e455b5ef5b88 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:27 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:27.131410 11006 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:27.132293 11006 tablet_bootstrap.cc:492] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:27.132740 11006 ts_tablet_manager.cc:1367] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.009s
I20240503 14:13:27.133862 11006 raft_consensus.cc:348] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:27.134332 11006 raft_consensus.cc:729] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:27.134696 11006 consensus_queue.cc:260] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:27.120457 6655 catalog_manager.cc:5484] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:27.137367 11006 ts_tablet_manager.cc:1398] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 14:13:27.138574 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-d4d6c0c0b6c249208e67400face0b9a6 received from {username='slave'} at 127.0.186.2:44207
I20240503 14:13:27.138803 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-d4d6c0c0b6c249208e67400face0b9a6 on tablet d4d6c0c0b6c249208e67400face0b9a6 with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:27.139309 11038 tablet_replica.cc:330] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:27.140888 11038 raft_consensus.cc:2229] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.150702 7059 consensus_queue.cc:237] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:27.151634 9929 raft_consensus.cc:1203] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:27.152385 11041 tablet_replica.cc:330] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:27.152642 11041 raft_consensus.cc:2229] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.153170 9798 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:27.154877 11038 raft_consensus.cc:2258] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.154924 10371 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:27.155498 11042 tablet_replica.cc:330] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:27.157014 11038 ts_tablet_manager.cc:1875] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.152959 11041 raft_consensus.cc:2258] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.159102 6798 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:27.159785 10785 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:27.162508 10371 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:27.165506 11041 ts_tablet_manager.cc:1875] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:27.164460 11042 raft_consensus.cc:2229] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.165881 11042 raft_consensus.cc:2258] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.164837 10828 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.166133 10828 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:27.166829 10828 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:27.167991 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458ebc1295ee49999586eb699a29bbf8" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:27.168216 7191 raft_consensus.cc:2454] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:27.168819 10696 raft_consensus.cc:482] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.168988 9600 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:27.169003 10696 raft_consensus.cc:504] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.169550 10905 raft_consensus.cc:2781] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.169657 10905 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.169708 10696 leader_election.cc:288] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:27.169746 10905 raft_consensus.cc:3037] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.170509 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca50346c04c427eb59e9f036a27dccc" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:27.170739 9532 raft_consensus.cc:2454] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:27.171543 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca50346c04c427eb59e9f036a27dccc" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:27.171792 9929 raft_consensus.cc:2454] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:27.172257 9733 leader_election.cc:302] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:27.172922 10696 raft_consensus.cc:2781] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.173010 10696 raft_consensus.cc:482] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.173121 10696 raft_consensus.cc:3037] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.175364 11041 ts_tablet_manager.cc:1888] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:27.175746 10696 raft_consensus.cc:504] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.176522 11042 ts_tablet_manager.cc:1875] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:27.178797 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458ebc1295ee49999586eb699a29bbf8" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:27.179116 6800 raft_consensus.cc:2454] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:27.179395 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca50346c04c427eb59e9f036a27dccc" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:27.179702 9929 raft_consensus.cc:3037] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.179965 11038 ts_tablet_manager.cc:1888] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:27.180114 11038 log.cc:1196] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/48831ea899c445e199c5e455b5ef5b88
I20240503 14:13:27.180564 11038 ts_tablet_manager.cc:1909] T 48831ea899c445e199c5e455b5ef5b88 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:27.181429 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca50346c04c427eb59e9f036a27dccc" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:27.181558 11041 log.cc:1196] T b556ba3bf82b40d988c0173f742a0ff2 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/b556ba3bf82b40d988c0173f742a0ff2
I20240503 14:13:27.181651 9532 raft_consensus.cc:3037] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.182381 11041 tablet_replica.cc:330] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:27.182606 9929 raft_consensus.cc:2454] T 5ca50346c04c427eb59e9f036a27dccc P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:27.182672 11041 raft_consensus.cc:2229] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.182997 11041 raft_consensus.cc:2258] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.183127 9733 leader_election.cc:302] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:27.183460 10698 raft_consensus.cc:2781] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.183627 10698 raft_consensus.cc:686] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:27.183822 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet b556ba3bf82b40d988c0173f742a0ff2 (table rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]) successfully deleted
I20240503 14:13:27.183987 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 48831ea899c445e199c5e455b5ef5b88 (table rebalancer_dynamic_table_new_name [id=3eda6fb4810644cfa87f453aa2b76888]) successfully deleted
I20240503 14:13:27.183908 10698 consensus_queue.cc:237] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.184075 11041 ts_tablet_manager.cc:1875] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:27.185827 10696 leader_election.cc:288] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:27.187379 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:27.188333 10905 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:27.190938 11041 ts_tablet_manager.cc:1888] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:27.191217 11041 log.cc:1196] T 0b18e1d58117421c82ba86c08cd1253b P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/0b18e1d58117421c82ba86c08cd1253b
I20240503 14:13:27.191733 9532 raft_consensus.cc:2454] T 5ca50346c04c427eb59e9f036a27dccc P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:27.192080 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 0b18e1d58117421c82ba86c08cd1253b (table rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]) successfully deleted
I20240503 14:13:27.194140 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458ebc1295ee49999586eb699a29bbf8" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:27.194515 6800 raft_consensus.cc:3037] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.197192 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0473ccc3b6478d809567e3b681c151 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796]), partition=
I20240503 14:13:27.197574 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0473ccc3b6478d809567e3b681c151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.198859 6800 raft_consensus.cc:2454] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:27.199319 9599 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:27.199819 10828 raft_consensus.cc:2781] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.199950 10828 raft_consensus.cc:686] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:27.200234 10828 consensus_queue.cc:237] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:27.200615 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0473ccc3b6478d809567e3b681c151 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796]), partition=
I20240503 14:13:27.200989 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0473ccc3b6478d809567e3b681c151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.203805 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3160f1a600604f258f76614f13429563 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:27.204478 10905 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:27.205062 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 0c0473ccc3b6478d809567e3b681c151 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796]), partition=
I20240503 14:13:27.205021 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458ebc1295ee49999586eb699a29bbf8" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:27.205243 7191 raft_consensus.cc:3037] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.205417 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c0473ccc3b6478d809567e3b681c151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.208098 7191 raft_consensus.cc:2454] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:27.209497 6798 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:27.211977 9798 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:27.213963 6658 catalog_manager.cc:5484] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:27.218858 10986 tablet_bootstrap.cc:492] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:27.220521 10986 tablet_bootstrap.cc:654] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.222600 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 3160f1a600604f258f76614f13429563 with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:27.235953 9872 raft_consensus.cc:1050] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:27.236845 9872 consensus_queue.cc:237] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:27.238152 10987 tablet_bootstrap.cc:492] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:27.239812 10987 tablet_bootstrap.cc:654] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.246308 9929 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.246768 10845 tablet_bootstrap.cc:492] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:27.248782 10845 tablet_bootstrap.cc:654] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.250672 10729 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.253577 6655 catalog_manager.cc:5484] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:27.254017 7059 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.254791 10729 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.262971 6658 catalog_manager.cc:5484] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:27.194893 11042 ts_tablet_manager.cc:1888] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:27.266963 11042 log.cc:1196] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/48831ea899c445e199c5e455b5ef5b88
I20240503 14:13:27.268853 7191 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.269531 10848 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.273208 11042 ts_tablet_manager.cc:1909] T 48831ea899c445e199c5e455b5ef5b88 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:27.277894 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 48831ea899c445e199c5e455b5ef5b88 (table rebalancer_dynamic_table_new_name [id=3eda6fb4810644cfa87f453aa2b76888]) successfully deleted
I20240503 14:13:27.278988 11048 tablet_replica.cc:330] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:27.281724 11048 raft_consensus.cc:2229] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.282034 11048 raft_consensus.cc:2258] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.282195 10986 tablet_bootstrap.cc:492] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:27.282397 10986 ts_tablet_manager.cc:1367] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.013s
I20240503 14:13:27.284471 11048 ts_tablet_manager.cc:1875] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:27.284713 10986 raft_consensus.cc:348] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.285046 10986 raft_consensus.cc:374] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.285285 10986 raft_consensus.cc:729] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.285729 10986 consensus_queue.cc:260] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.291302 10986 ts_tablet_manager.cc:1398] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:13:27.298708 11048 ts_tablet_manager.cc:1888] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:27.298941 11048 log.cc:1196] T 3160f1a600604f258f76614f13429563 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/3160f1a600604f258f76614f13429563
I20240503 14:13:27.299906 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 3160f1a600604f258f76614f13429563 (table rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]) successfully deleted
I20240503 14:13:27.306640 10848 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.311347 9930 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.318290 6658 catalog_manager.cc:5484] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.320284 7191 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.329249 10845 tablet_bootstrap.cc:492] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:27.330085 10987 tablet_bootstrap.cc:492] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:27.330767 7059 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.331727 10845 ts_tablet_manager.cc:1367] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.005s
I20240503 14:13:27.332022 10987 ts_tablet_manager.cc:1367] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.094s user 0.006s sys 0.004s
I20240503 14:13:27.334358 10845 raft_consensus.cc:348] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.342890 10845 raft_consensus.cc:374] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.333137 10987 raft_consensus.cc:348] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.343367 10987 raft_consensus.cc:374] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.343539 10845 raft_consensus.cc:729] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.343740 10987 raft_consensus.cc:729] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.344162 10845 consensus_queue.cc:260] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.344789 10987 consensus_queue.cc:260] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.349426 10845 ts_tablet_manager.cc:1398] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 14:13:27.349879 10987 ts_tablet_manager.cc:1398] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.018s user 0.002s sys 0.001s
I20240503 14:13:27.373651 10985 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.389976 11011 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.464079 6930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e431272976034209b2c40cb0e460e17a"
dest_uuid: "7186ed8510424fd597173eaf10c57b12"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60394
I20240503 14:13:27.464344 6930 raft_consensus.cc:595] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:27.484983 6930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3"
dest_uuid: "7186ed8510424fd597173eaf10c57b12"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60408
I20240503 14:13:27.485247 6930 raft_consensus.cc:595] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:27.488600 9872 raft_consensus.cc:979] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: : Instructing follower ea801601713a4d55aee0de9e3473e0bc to start an election
I20240503 14:13:27.488790 10848 raft_consensus.cc:1067] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Signalling peer ea801601713a4d55aee0de9e3473e0bc to start an election
I20240503 14:13:27.490056 11040 raft_consensus.cc:1050] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:27.490097 9929 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
from {username='slave'} at 127.0.186.2:44207
I20240503 14:13:27.490363 9929 raft_consensus.cc:482] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:27.490515 9929 raft_consensus.cc:3037] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:27.490976 11040 consensus_queue.cc:237] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:27.496677 10201 raft_consensus.cc:1050] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:27.497107 6930 raft_consensus.cc:1203] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:27.497705 10201 consensus_queue.cc:237] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:27.495606 9929 raft_consensus.cc:504] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:27.504989 6930 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.505081 7191 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.505807 10785 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.506186 10371 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.506433 9665 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.507040 10371 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.509023 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:27.509322 6931 raft_consensus.cc:3032] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:27.509449 6931 raft_consensus.cc:729] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:27.509845 6931 consensus_queue.cc:260] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:27.510445 6931 raft_consensus.cc:3037] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:27.510882 9929 leader_election.cc:288] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:27.514389 6931 raft_consensus.cc:2454] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:27.529278 6931 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:27.528259 11040 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:27.533097 9665 raft_consensus.cc:1261] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.533776 10988 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.535322 11121 raft_consensus.cc:482] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.535570 11121 raft_consensus.cc:504] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.536374 6929 raft_consensus.cc:1261] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.536728 11121 leader_election.cc:288] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:27.536672 9665 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:27.537689 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0473ccc3b6478d809567e3b681c151" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:27.537911 9663 raft_consensus.cc:2454] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:27.537955 7059 raft_consensus.cc:1261] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.538705 9863 leader_election.cc:302] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:27.539093 10988 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.539714 10988 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.545385 11121 raft_consensus.cc:2781] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.545542 11121 raft_consensus.cc:482] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.545632 11121 raft_consensus.cc:3037] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.545529 6658 catalog_manager.cc:5484] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:27.562660 11121 raft_consensus.cc:504] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.563963 11121 leader_election.cc:288] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:27.565202 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0473ccc3b6478d809567e3b681c151" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:27.565425 9798 raft_consensus.cc:2454] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:27.564188 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0473ccc3b6478d809567e3b681c151" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:27.565883 9665 raft_consensus.cc:3037] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.569202 9665 raft_consensus.cc:2454] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:27.569577 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c0473ccc3b6478d809567e3b681c151" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:27.569828 9798 raft_consensus.cc:3037] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.573654 9798 raft_consensus.cc:2454] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:27.575264 9864 leader_election.cc:302] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:27.575783 11121 raft_consensus.cc:2781] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:27.575945 11121 raft_consensus.cc:686] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:27.576260 11121 consensus_queue.cc:237] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:27.583863 11130 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:27.588256 7059 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:27.594036 6654 catalog_manager.cc:5484] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:27.594173 7191 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:27.600170 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:27.600423 6798 raft_consensus.cc:3037] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:27.601760 6655 catalog_manager.cc:5484] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:27.603340 6798 raft_consensus.cc:2454] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:27.603927 9863 leader_election.cc:302] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:27.604372 11121 raft_consensus.cc:2781] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:27.604560 11121 raft_consensus.cc:686] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:27.605386 11121 consensus_queue.cc:237] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
W20240503 14:13:27.608531 7125 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 46001.7us
I20240503 14:13:27.610379 6929 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:27.619910 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "343520037eac42c4ac76fa1cc5e26fc3" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:27.620175 7190 raft_consensus.cc:3037] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:27.622714 7190 raft_consensus.cc:2454] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:27.624493 6655 catalog_manager.cc:5484] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 1 to 2, leader changed from 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:27.632566 10698 consensus_queue.cc:1035] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.652184 11009 consensus_queue.cc:1035] T 5ca50346c04c427eb59e9f036a27dccc P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.657212 10828 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.679311 10250 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.716800 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:27.722997 9797 raft_consensus.cc:1261] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:27.723208 9664 raft_consensus.cc:1261] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:27.723862 11121 consensus_queue.cc:1035] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.724689 11121 consensus_queue.cc:1035] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:27.728621 11062 raft_consensus.cc:1050] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:27.729486 11062 consensus_queue.cc:237] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:27.758796 10848 raft_consensus.cc:979] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: : Instructing follower ea801601713a4d55aee0de9e3473e0bc to start an election
I20240503 14:13:27.759008 10848 raft_consensus.cc:1067] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Signalling peer ea801601713a4d55aee0de9e3473e0bc to start an election
I20240503 14:13:27.759790 9930 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e431272976034209b2c40cb0e460e17a"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
from {username='slave'} at 127.0.186.2:44207
I20240503 14:13:27.760048 9930 raft_consensus.cc:482] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:27.760171 9930 raft_consensus.cc:3037] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:27.761235 6658 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 0c0473ccc3b6478d809567e3b681c151 (table rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796]). Expected version 1 got 0
I20240503 14:13:27.763978 9930 raft_consensus.cc:504] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:27.765194 9930 leader_election.cc:288] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:27.765895 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e431272976034209b2c40cb0e460e17a" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:27.766021 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e431272976034209b2c40cb0e460e17a" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:27.766146 7190 raft_consensus.cc:3037] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:27.766234 7059 raft_consensus.cc:3037] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:27.766582 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e431272976034209b2c40cb0e460e17a" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:27.766772 6930 raft_consensus.cc:3032] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:27.766983 6930 raft_consensus.cc:729] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:27.767244 6930 consensus_queue.cc:260] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:27.767707 6930 raft_consensus.cc:3037] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:27.768141 7060 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.770124 7191 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.770220 7059 raft_consensus.cc:2454] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:27.770795 11062 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.771401 11062 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.771800 9930 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:27.772361 6930 raft_consensus.cc:2454] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:27.772681 10729 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:27.773761 9863 leader_election.cc:302] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:27.773772 7190 raft_consensus.cc:2454] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:27.774116 10933 raft_consensus.cc:2781] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:27.774279 10933 raft_consensus.cc:686] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:27.774626 10933 consensus_queue.cc:237] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:27.781790 6655 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 0c0473ccc3b6478d809567e3b681c151 (table rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796]). Expected version 1 got 0
I20240503 14:13:27.788995 11159 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:27.791491 11158 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:27.792328 11159 tablet.cc:1620] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:27.792273 11062 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.793339 11158 tablet.cc:1620] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:27.798156 7191 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.799736 9929 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.800558 11163 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:27.803705 7060 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.802826 6655 catalog_manager.cc:5484] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:27.808813 11163 tablet.cc:1620] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:27.807511 6651 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 0c0473ccc3b6478d809567e3b681c151 (table rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796]). Expected version 1 got 0
I20240503 14:13:27.811986 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:27.822377 6930 consensus_queue.cc:237] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:27.824484 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 1de841c504fc4bd78ea1a6a32bd34aeb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.824924 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de841c504fc4bd78ea1a6a32bd34aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.825155 7060 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:27.825181 9930 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:27.825731 10848 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:27.826561 11062 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:27.828016 9643 tablet_service.cc:1467] Processing CreateTablet for tablet 84b58d63957848d7a8b0eae9f60bbbf1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.828420 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b58d63957848d7a8b0eae9f60bbbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.829878 11062 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.833588 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 84b58d63957848d7a8b0eae9f60bbbf1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.833899 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 1de841c504fc4bd78ea1a6a32bd34aeb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.834079 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b58d63957848d7a8b0eae9f60bbbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.834355 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de841c504fc4bd78ea1a6a32bd34aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.834708 9929 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.837347 7039 tablet_service.cc:1467] Processing CreateTablet for tablet c59daa2d12f446899c7bb9970e1694da (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.837531 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 84b58d63957848d7a8b0eae9f60bbbf1 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.837771 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59daa2d12f446899c7bb9970e1694da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.837858 9645 tablet_service.cc:1467] Processing CreateTablet for tablet c59daa2d12f446899c7bb9970e1694da (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.838243 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59daa2d12f446899c7bb9970e1694da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.838713 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b58d63957848d7a8b0eae9f60bbbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.847544 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 34f1ed0cfa8948048b3762919481367c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:27.848789 7060 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:27.851824 11168 tablet_bootstrap.cc:492] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:27.853031 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 1de841c504fc4bd78ea1a6a32bd34aeb (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.853274 9778 tablet_service.cc:1467] Processing CreateTablet for tablet c59daa2d12f446899c7bb9970e1694da (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:27.853560 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1de841c504fc4bd78ea1a6a32bd34aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.854030 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59daa2d12f446899c7bb9970e1694da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:27.859661 6655 catalog_manager.cc:5484] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:27.868872 11168 tablet_bootstrap.cc:654] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.872072 11169 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:27.874480 11169 tablet_bootstrap.cc:654] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.875784 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796] alter complete (version 1)
I20240503 14:13:27.876410 6658 catalog_manager.cc:5484] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 1 to 2, leader changed from 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:27.878382 11166 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:27.881114 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f1ed0cfa8948048b3762919481367c with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:27.882218 6655 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 0c0473ccc3b6478d809567e3b681c151 (table rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796]). Expected version 1 got 0
I20240503 14:13:27.885397 11171 tablet_bootstrap.cc:492] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:27.887854 11166 tablet_bootstrap.cc:654] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.889605 11171 tablet_bootstrap.cc:654] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.892791 11174 tablet_replica.cc:330] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:27.895494 11169 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:27.895702 11169 ts_tablet_manager.cc:1367] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.000s
I20240503 14:13:27.896495 11169 raft_consensus.cc:348] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:27.896690 11169 raft_consensus.cc:374] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.896780 11169 raft_consensus.cc:729] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.897034 11169 consensus_queue.cc:260] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:27.897603 11169 ts_tablet_manager.cc:1398] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:27.898502 11174 raft_consensus.cc:2229] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:27.898896 11174 raft_consensus.cc:2258] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:27.899720 11168 tablet_bootstrap.cc:492] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:27.900056 11168 ts_tablet_manager.cc:1367] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.000s
I20240503 14:13:27.900736 11174 ts_tablet_manager.cc:1875] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:27.901566 11168 raft_consensus.cc:348] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:27.904553 11168 raft_consensus.cc:374] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.904772 11168 raft_consensus.cc:729] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.905082 11168 consensus_queue.cc:260] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:27.909845 11168 ts_tablet_manager.cc:1398] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:27.911733 11167 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:27.913218 11168 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:27.915055 11168 tablet_bootstrap.cc:654] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.915933 11167 tablet_bootstrap.cc:654] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.917243 11166 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:27.917471 11166 ts_tablet_manager.cc:1367] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.000s
I20240503 14:13:27.918361 11166 raft_consensus.cc:348] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:27.918702 11166 raft_consensus.cc:374] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.918846 11166 raft_consensus.cc:729] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.919152 11166 consensus_queue.cc:260] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:27.919289 11174 ts_tablet_manager.cc:1888] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:27.920516 11174 log.cc:1196] T 34f1ed0cfa8948048b3762919481367c P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/34f1ed0cfa8948048b3762919481367c
I20240503 14:13:27.921536 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 34f1ed0cfa8948048b3762919481367c (table rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]) successfully deleted
I20240503 14:13:27.926616 11167 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:27.926857 11167 ts_tablet_manager.cc:1367] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.002s
I20240503 14:13:27.929343 11167 raft_consensus.cc:348] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:27.929728 11167 raft_consensus.cc:374] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.929893 11167 raft_consensus.cc:729] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.930243 11167 consensus_queue.cc:260] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:27.930950 11167 ts_tablet_manager.cc:1398] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:27.933714 11168 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:27.933952 11168 ts_tablet_manager.cc:1367] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.006s
I20240503 14:13:27.934793 11166 ts_tablet_manager.cc:1398] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20240503 14:13:27.934815 11168 raft_consensus.cc:348] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:27.935146 11168 raft_consensus.cc:374] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.935827 11168 raft_consensus.cc:729] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.936110 11168 consensus_queue.cc:260] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:27.936422 11166 tablet_bootstrap.cc:492] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:27.936672 11168 ts_tablet_manager.cc:1398] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:27.937889 11168 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:27.940323 11121 raft_consensus.cc:1050] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:27.941073 11166 tablet_bootstrap.cc:654] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.941448 11168 tablet_bootstrap.cc:654] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.941360 11121 consensus_queue.cc:237] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:27.947515 11171 tablet_bootstrap.cc:492] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:27.948194 11171 ts_tablet_manager.cc:1367] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.011s
I20240503 14:13:27.948554 11168 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:27.948758 11168 ts_tablet_manager.cc:1367] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 14:13:27.949851 11168 raft_consensus.cc:348] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:27.950119 11168 raft_consensus.cc:374] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.950295 11168 raft_consensus.cc:729] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.950822 11168 consensus_queue.cc:260] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:27.951086 11171 raft_consensus.cc:348] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:27.951208 6930 raft_consensus.cc:1261] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:27.951287 11171 raft_consensus.cc:374] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.951375 11171 raft_consensus.cc:729] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.951647 11171 consensus_queue.cc:260] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:27.951833 11118 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:27.952281 11171 ts_tablet_manager.cc:1398] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:27.952545 11171 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:27.952538 11168 ts_tablet_manager.cc:1398] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:27.954252 11171 tablet_bootstrap.cc:654] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:27.955962 11166 tablet_bootstrap.cc:492] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:27.956670 7060 raft_consensus.cc:1261] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:27.956972 9797 raft_consensus.cc:1261] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:27.957796 11166 ts_tablet_manager.cc:1367] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240503 14:13:27.957885 11118 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:27.958413 11171 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:27.958477 11118 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:27.959535 11166 raft_consensus.cc:348] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:27.960084 11171 ts_tablet_manager.cc:1367] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240503 14:13:27.961153 11171 raft_consensus.cc:348] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:27.963218 11166 raft_consensus.cc:374] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.963512 11166 raft_consensus.cc:729] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.963830 11171 raft_consensus.cc:374] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:27.963909 11171 raft_consensus.cc:729] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:27.964135 11171 consensus_queue.cc:260] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:27.964570 11171 ts_tablet_manager.cc:1398] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240503 14:13:27.964929 11166 consensus_queue.cc:260] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:27.965864 11166 ts_tablet_manager.cc:1398] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:27.978986 10933 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:27.979626 9797 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:27.985285 6658 catalog_manager.cc:5484] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:27.985502 7060 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:27.987394 6930 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:27.993809 9930 consensus_queue.cc:237] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:27.994514 10785 raft_consensus.cc:482] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:27.994683 10785 raft_consensus.cc:504] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:27.995374 10785 leader_election.cc:288] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:27.995946 6929 raft_consensus.cc:1261] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:27.995880 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59daa2d12f446899c7bb9970e1694da" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:27.996111 9797 raft_consensus.cc:2454] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:27.996481 6996 leader_election.cc:302] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 3efdbd3506274dba846ae40404f0572c; no voters:
I20240503 14:13:27.996747 10785 raft_consensus.cc:2781] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:27.996659 11118 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:27.996860 10785 raft_consensus.cc:482] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:27.996943 10785 raft_consensus.cc:3037] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:27.997166 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59daa2d12f446899c7bb9970e1694da" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:27.997435 9665 raft_consensus.cc:2454] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:27.999850 7060 raft_consensus.cc:1261] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:27.999472 10785 raft_consensus.cc:504] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:27.999874 11118 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:28.000174 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:28.000906 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59daa2d12f446899c7bb9970e1694da" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:28.001111 9665 raft_consensus.cc:3037] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.000982 6929 consensus_queue.cc:237] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:28.003808 9665 raft_consensus.cc:2454] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:28.003793 7190 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.004256 6995 leader_election.cc:302] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:28.004451 10848 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.004565 10371 raft_consensus.cc:2781] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.004748 10371 raft_consensus.cc:686] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:28.005029 10371 consensus_queue.cc:237] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:28.005465 10785 leader_election.cc:288] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:28.005692 10933 consensus_queue.cc:1035] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.005764 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59daa2d12f446899c7bb9970e1694da" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:28.005986 9797 raft_consensus.cc:3037] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:28.006161 6867 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3e2d7664e6c45d69f4f8b20dcf0e4d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.007190 7060 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.008024 7059 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:28.008942 9797 raft_consensus.cc:2454] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:28.009495 10848 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.019716 11062 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.019570 6929 raft_consensus.cc:2932] T d4d6c0c0b6c249208e67400face0b9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:28.021968 11159 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:28.022545 11159 tablet.cc:1620] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:28.026311 6650 catalog_manager.cc:5484] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:28.027511 7191 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.026336 6655 catalog_manager.cc:5484] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:28.037171 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d6c0c0b6c249208e67400face0b9a6 with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:28.038126 10698 raft_consensus.cc:482] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:28.038630 10698 raft_consensus.cc:504] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:28.040310 10698 leader_election.cc:288] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:28.041043 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d4d6c0c0b6c249208e67400face0b9a6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:28.041904 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de841c504fc4bd78ea1a6a32bd34aeb" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:28.042178 9532 raft_consensus.cc:2454] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:28.042994 9735 leader_election.cc:302] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:28.043373 11163 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:28.043704 10698 raft_consensus.cc:2781] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:28.043805 10698 raft_consensus.cc:482] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:28.043895 11163 tablet.cc:1620] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:28.043911 10698 raft_consensus.cc:3037] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.046862 10698 raft_consensus.cc:504] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:28.047547 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de841c504fc4bd78ea1a6a32bd34aeb" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:28.047784 9665 raft_consensus.cc:2454] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:28.047737 11158 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:28.050477 11158 tablet.cc:1620] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:28.050735 7059 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.059854 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de841c504fc4bd78ea1a6a32bd34aeb" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:28.060120 9532 raft_consensus.cc:3037] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.060202 10698 leader_election.cc:288] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:28.062826 9532 raft_consensus.cc:2454] T 1de841c504fc4bd78ea1a6a32bd34aeb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:28.063313 9735 leader_election.cc:302] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:28.063824 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1de841c504fc4bd78ea1a6a32bd34aeb" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:28.064054 9665 raft_consensus.cc:3037] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.064579 10698 raft_consensus.cc:2781] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.064786 10698 raft_consensus.cc:686] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:28.065044 10698 consensus_queue.cc:237] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:28.067157 9665 raft_consensus.cc:2454] T 1de841c504fc4bd78ea1a6a32bd34aeb P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:28.070544 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=7ed17cf89ebc46ef9d26b2e5248dc796] alter complete (version 2)
I20240503 14:13:28.071214 6651 catalog_manager.cc:5484] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:28.075400 6655 catalog_manager.cc:5484] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:28.079254 10933 raft_consensus.cc:482] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:28.079475 10933 raft_consensus.cc:504] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.080436 10933 leader_election.cc:288] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:28.081823 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b58d63957848d7a8b0eae9f60bbbf1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:28.082113 9532 raft_consensus.cc:2454] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:28.082041 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b58d63957848d7a8b0eae9f60bbbf1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:28.082620 9865 leader_election.cc:302] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:28.083534 10933 raft_consensus.cc:2781] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:28.083688 10933 raft_consensus.cc:482] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:28.083775 10933 raft_consensus.cc:3037] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.087364 10933 raft_consensus.cc:504] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.090834 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b58d63957848d7a8b0eae9f60bbbf1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:28.091127 9532 raft_consensus.cc:3037] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.092118 7190 consensus_queue.cc:237] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:28.094622 7059 raft_consensus.cc:1261] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.094769 6798 raft_consensus.cc:1261] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:28.094835 7126 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 847ef240dcc64fb794ffcc06e483dc08. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.095279 10201 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.095620 11132 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.098875 7190 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:28.099635 11121 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:28.102769 10988 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.103526 6798 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.112409 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b58d63957848d7a8b0eae9f60bbbf1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:28.113802 10933 leader_election.cc:288] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:28.082599 9665 raft_consensus.cc:2454] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:28.116619 9532 raft_consensus.cc:2454] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:28.117527 9865 leader_election.cc:302] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:28.118257 11197 tablet_replica.cc:330] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:28.118680 10933 raft_consensus.cc:2781] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.118819 11197 raft_consensus.cc:2229] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.118942 10933 raft_consensus.cc:686] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:28.119468 10933 consensus_queue.cc:237] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.125247 6798 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:28.129686 10933 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:28.132972 11197 raft_consensus.cc:2258] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.138835 11197 ts_tablet_manager.cc:1875] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:28.139897 7060 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.146886 6657 catalog_manager.cc:5484] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:28.148036 6654 catalog_manager.cc:5484] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.149886 11197 ts_tablet_manager.cc:1888] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:28.150535 11197 log.cc:1196] T d4d6c0c0b6c249208e67400face0b9a6 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/d4d6c0c0b6c249208e67400face0b9a6
I20240503 14:13:28.151433 9665 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:28.153822 6930 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:28.155700 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet d4d6c0c0b6c249208e67400face0b9a6 (table rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]) successfully deleted
I20240503 14:13:28.155679 10933 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:28.159210 9930 consensus_queue.cc:237] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:28.161623 6798 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:28.161715 6929 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:28.162849 7190 raft_consensus.cc:1261] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:28.162936 11195 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:28.163125 11118 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:28.163697 11118 consensus_queue.cc:1035] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:28.167279 11195 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:28.168115 7190 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:28.174749 6655 catalog_manager.cc:5484] T 343520037eac42c4ac76fa1cc5e26fc3 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:28.178289 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 343520037eac42c4ac76fa1cc5e26fc3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:28.178936 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 343520037eac42c4ac76fa1cc5e26fc3 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:28.179543 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:28.179726 11214 tablet_replica.cc:330] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:28.183116 6800 raft_consensus.cc:2932] T 343520037eac42c4ac76fa1cc5e26fc3 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:28.186585 11214 raft_consensus.cc:2229] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.186928 11214 raft_consensus.cc:2258] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.188753 11214 ts_tablet_manager.cc:1875] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:28.196177 11159 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:28.197101 11158 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:28.202360 11214 ts_tablet_manager.cc:1888] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:13:28.202584 11214 log.cc:1196] T 343520037eac42c4ac76fa1cc5e26fc3 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/343520037eac42c4ac76fa1cc5e26fc3
I20240503 14:13:28.203416 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 343520037eac42c4ac76fa1cc5e26fc3 (table rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]) successfully deleted
I20240503 14:13:28.203649 11163 tablet.cc:1722] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:28.215163 7190 consensus_queue.cc:237] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:28.217576 6930 raft_consensus.cc:1261] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:28.218230 11132 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:28.219324 6657 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=7ed17cf89ebc46ef9d26b2e5248dc796] alter complete (version 3)
I20240503 14:13:28.220714 9663 raft_consensus.cc:1261] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:28.222189 10988 consensus_queue.cc:1035] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:28.222748 10988 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:28.225163 6930 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:28.227535 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b8014635e8294062b1c1985a100307ff with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:28.229802 6658 catalog_manager.cc:5484] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:28.234714 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet b8014635e8294062b1c1985a100307ff with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:28.235677 11217 tablet_replica.cc:330] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:28.235949 11217 raft_consensus.cc:2229] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.236335 11217 raft_consensus.cc:2258] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.236981 9664 raft_consensus.cc:2932] T b8014635e8294062b1c1985a100307ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:28.239725 11217 ts_tablet_manager.cc:1875] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:28.246268 7190 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:28.247776 11121 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:28.255743 7060 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:28.257491 11118 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:28.268180 11217 ts_tablet_manager.cc:1888] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:28.268369 11217 log.cc:1196] T b8014635e8294062b1c1985a100307ff P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/b8014635e8294062b1c1985a100307ff
I20240503 14:13:28.270994 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet b8014635e8294062b1c1985a100307ff (table rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]) successfully deleted
I20240503 14:13:28.298959 9797 consensus_queue.cc:237] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:28.309355 9532 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:28.309638 9930 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:28.310249 11009 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:28.310808 10698 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:28.311079 9735 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01e27516acca49459176362caf73595e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.320600 11009 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:28.321991 6654 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:28.322196 6654 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:28.324095 9929 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:28.325834 6654 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 0c0473ccc3b6478d809567e3b681c151
I20240503 14:13:28.326761 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0473ccc3b6478d809567e3b681c151 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:28.327037 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0473ccc3b6478d809567e3b681c151 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:28.327829 11227 tablet_replica.cc:330] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:28.328042 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c0473ccc3b6478d809567e3b681c151 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:28 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:28.328146 11227 raft_consensus.cc:2229] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.328344 11197 tablet_replica.cc:330] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:28.328378 11227 raft_consensus.cc:2258] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.328536 11197 raft_consensus.cc:2229] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.328758 11197 raft_consensus.cc:2258] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.330389 11197 ts_tablet_manager.cc:1875] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.336576 11227 ts_tablet_manager.cc:1875] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.336972 6654 catalog_manager.cc:5484] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:28.337450 11197 ts_tablet_manager.cc:1888] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:28.337600 11197 log.cc:1196] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/0c0473ccc3b6478d809567e3b681c151
I20240503 14:13:28.337985 11197 ts_tablet_manager.cc:1909] T 0c0473ccc3b6478d809567e3b681c151 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:28.339177 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 0c0473ccc3b6478d809567e3b681c151 (table rebalancer_dynamic_table_new_name [id=7ed17cf89ebc46ef9d26b2e5248dc796]) successfully deleted
I20240503 14:13:28.340217 11226 tablet_replica.cc:330] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:28.340597 11226 raft_consensus.cc:2229] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:28.341048 11226 raft_consensus.cc:2258] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.351555 11227 ts_tablet_manager.cc:1888] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:28.351840 11227 log.cc:1196] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/0c0473ccc3b6478d809567e3b681c151
I20240503 14:13:28.352437 11227 ts_tablet_manager.cc:1909] T 0c0473ccc3b6478d809567e3b681c151 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:28.353859 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 0c0473ccc3b6478d809567e3b681c151 (table rebalancer_dynamic_table_new_name [id=7ed17cf89ebc46ef9d26b2e5248dc796]) successfully deleted
I20240503 14:13:28.352303 9532 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:28.355106 11226 ts_tablet_manager.cc:1875] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:28.364380 11226 ts_tablet_manager.cc:1888] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:28.364650 11226 log.cc:1196] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/0c0473ccc3b6478d809567e3b681c151
I20240503 14:13:28.365180 11226 ts_tablet_manager.cc:1909] T 0c0473ccc3b6478d809567e3b681c151 P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:28.366657 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 0c0473ccc3b6478d809567e3b681c151 (table rebalancer_dynamic_table_new_name [id=7ed17cf89ebc46ef9d26b2e5248dc796]) successfully deleted
I20240503 14:13:28.376581 7060 consensus_queue.cc:237] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:28.379106 7190 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:28.379204 6995 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e0bb82edac2464b9d618c1c80807fb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.379307 6800 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.379839 10371 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.380162 10785 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.383903 6800 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:28.384574 10785 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:28.388804 7190 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:28.390197 6658 catalog_manager.cc:5484] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:28.399950 11238 ts_tablet_manager.cc:902] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:28.404244 11238 tablet_copy_client.cc:308] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:28.404942 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:46745)
I20240503 14:13:28.405074 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:46745: session id = 5b7e36bf354f480c90859cb8867b59cf-b3e2d7664e6c45d69f4f8b20dcf0e4d0
I20240503 14:13:28.407227 6951 tablet_copy_source_session.cc:205] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:28.408442 11238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.413628 11238 tablet_copy_client.cc:783] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:28.413889 11238 tablet_copy_client.cc:647] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:28.416424 6929 consensus_queue.cc:237] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:28.417747 11238 tablet_copy_client.cc:523] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:28.418843 7190 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.419255 7060 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:28.419763 6866 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c727d4554464442a6a81a8ad4996c84. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.419931 10729 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.420634 10729 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.421463 11238 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:28.422956 9872 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.423689 7190 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.428843 7060 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.428925 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:28.430806 6651 catalog_manager.cc:5484] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.441987 11040 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.449826 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 48198af7614b48a2afaa3bb95e071ed3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c]), partition=
I20240503 14:13:28.450030 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 48198af7614b48a2afaa3bb95e071ed3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c]), partition=
I20240503 14:13:28.450286 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48198af7614b48a2afaa3bb95e071ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.450412 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48198af7614b48a2afaa3bb95e071ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.450425 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 48198af7614b48a2afaa3bb95e071ed3 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c]), partition=
I20240503 14:13:28.450745 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48198af7614b48a2afaa3bb95e071ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.454228 10785 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.474691 11255 tablet_bootstrap.cc:492] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:28.475476 11225 consensus_queue.cc:1035] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.481381 11255 tablet_bootstrap.cc:654] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.481279 11225 consensus_queue.cc:1035] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.482251 11254 tablet_bootstrap.cc:492] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:28.484515 11254 tablet_bootstrap.cc:654] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.491617 11238 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:28.492164 11238 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:28.492578 11238 ts_tablet_manager.cc:1367] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.071s user 0.054s sys 0.006s
I20240503 14:13:28.493567 11238 raft_consensus.cc:348] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:28.493918 11238 raft_consensus.cc:729] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:28.494191 11238 consensus_queue.cc:260] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:28.495170 11254 tablet_bootstrap.cc:492] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:28.496646 11238 ts_tablet_manager.cc:1398] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:28.496891 11254 ts_tablet_manager.cc:1367] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 14:13:28.497241 11255 tablet_bootstrap.cc:492] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:28.497442 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-b3e2d7664e6c45d69f4f8b20dcf0e4d0 received from {username='slave'} at 127.0.186.5:46745
I20240503 14:13:28.497622 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-b3e2d7664e6c45d69f4f8b20dcf0e4d0 on tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:28.498098 11254 raft_consensus.cc:348] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.498500 11255 ts_tablet_manager.cc:1367] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.003s
I20240503 14:13:28.499666 11254 raft_consensus.cc:374] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.500061 11255 raft_consensus.cc:348] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.500597 11254 raft_consensus.cc:729] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.500916 11254 consensus_queue.cc:260] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.501293 11255 raft_consensus.cc:374] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.501973 11254 ts_tablet_manager.cc:1398] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:28.502151 11255 raft_consensus.cc:729] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.502664 11255 consensus_queue.cc:260] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.504633 11255 ts_tablet_manager.cc:1398] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:13:28.505935 11261 tablet_bootstrap.cc:492] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:28.508495 11261 tablet_bootstrap.cc:654] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.511379 11261 tablet_bootstrap.cc:492] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:28.511698 11261 ts_tablet_manager.cc:1367] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240503 14:13:28.514266 11261 raft_consensus.cc:348] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.515936 11261 raft_consensus.cc:374] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.516068 11261 raft_consensus.cc:729] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.516300 11261 consensus_queue.cc:260] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.517118 11261 ts_tablet_manager.cc:1398] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:28.533576 6929 consensus_queue.cc:237] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:28.536031 7190 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.536166 7060 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.536934 9872 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.537248 9665 raft_consensus.cc:3037] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.537320 11062 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:28.536934 6867 consensus_peers.cc:485] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9027b8bfee0443aab7420df3f7257bcb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.540289 11062 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.541502 7060 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.543773 11118 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.546603 6654 catalog_manager.cc:5484] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:28.550096 7190 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.554826 11121 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.576211 7060 consensus_queue.cc:237] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:28.578687 6929 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.578725 7190 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.579259 10785 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.579572 10371 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:28.579963 6995 consensus_peers.cc:485] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a44ed92d65114bfda2e0f38054b17ada. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.582976 10371 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:28.583981 6930 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:28.592227 6654 catalog_manager.cc:5484] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:28.593647 7190 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:28.617256 7060 consensus_queue.cc:237] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
W20240503 14:13:28.620152 6996 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a90c8e47862945f1a873b17da14bb397. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.620385 6800 raft_consensus.cc:1261] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.621122 6930 raft_consensus.cc:1261] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.621631 11040 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.621996 10785 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.623659 10907 raft_consensus.cc:482] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:28.623819 10907 raft_consensus.cc:504] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.624593 10907 leader_election.cc:288] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:28.625280 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48198af7614b48a2afaa3bb95e071ed3" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:28.625516 9665 raft_consensus.cc:2454] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:28.626094 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48198af7614b48a2afaa3bb95e071ed3" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:28.626320 9929 raft_consensus.cc:2454] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:28.626857 9467 leader_election.cc:302] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:28.627712 10907 raft_consensus.cc:2781] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:28.627839 10907 raft_consensus.cc:482] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:28.627918 10907 raft_consensus.cc:3037] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.630916 10907 raft_consensus.cc:504] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.632196 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48198af7614b48a2afaa3bb95e071ed3" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:28.632432 9665 raft_consensus.cc:3037] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.635113 9665 raft_consensus.cc:2454] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:28.635815 9467 leader_election.cc:302] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:28.637292 11040 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.639787 11288 raft_consensus.cc:2781] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:28.640276 6800 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.647296 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48198af7614b48a2afaa3bb95e071ed3" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:28.647615 9929 raft_consensus.cc:3037] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:28.648650 11288 raft_consensus.cc:686] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:28.649426 11288 consensus_queue.cc:237] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:28.652129 9929 raft_consensus.cc:2454] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:28.653017 10907 leader_election.cc:288] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:28.658062 6650 catalog_manager.cc:5484] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.674316 6929 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:28.681663 6654 catalog_manager.cc:5484] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:28.684463 11292 ts_tablet_manager.cc:902] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:28.734162 11292 tablet_copy_client.cc:308] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:28.735131 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 847ef240dcc64fb794ffcc06e483dc08 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:36385)
I20240503 14:13:28.735383 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 847ef240dcc64fb794ffcc06e483dc08 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:36385: session id = 3efdbd3506274dba846ae40404f0572c-847ef240dcc64fb794ffcc06e483dc08
I20240503 14:13:28.736742 9929 consensus_queue.cc:237] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:28.739210 7190 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:28.739290 7211 tablet_copy_source_session.cc:205] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:28.740604 11195 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:28.742036 7060 raft_consensus.cc:1261] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:28.742810 11195 consensus_queue.cc:1035] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:28.742933 11292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 847ef240dcc64fb794ffcc06e483dc08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.745290 11195 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:28.748381 7060 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:28.749011 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:28.759454 11292 tablet_copy_client.cc:783] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:28.759850 11292 tablet_copy_client.cc:647] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:28.761831 11292 tablet_copy_client.cc:523] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:28.766261 11292 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:28.768644 11302 ts_tablet_manager.cc:902] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:28.766675 6654 catalog_manager.cc:5484] T e431272976034209b2c40cb0e460e17a P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:28.780835 7190 raft_consensus.cc:2932] T e431272976034209b2c40cb0e460e17a P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:28.784432 9530 raft_consensus.cc:1203] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:28.784727 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e431272976034209b2c40cb0e460e17a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:28.786801 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet e431272976034209b2c40cb0e460e17a with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:28.788172 7191 consensus_queue.cc:237] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:28.793309 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaca37078ac4c4eb7f0bb6df0654860 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.793813 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aaca37078ac4c4eb7f0bb6df0654860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.797137 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaca37078ac4c4eb7f0bb6df0654860 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.797154 6780 tablet_service.cc:1467] Processing CreateTablet for tablet f6f8dfcb870a4461b03415db9c33452e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.797588 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f8dfcb870a4461b03415db9c33452e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.797603 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aaca37078ac4c4eb7f0bb6df0654860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.797540 9644 tablet_service.cc:1467] Processing CreateTablet for tablet f6f8dfcb870a4461b03415db9c33452e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.797925 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f8dfcb870a4461b03415db9c33452e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.799937 9643 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.799937 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.800303 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.800326 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.803848 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 8aaca37078ac4c4eb7f0bb6df0654860 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.804240 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aaca37078ac4c4eb7f0bb6df0654860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.808408 9910 tablet_service.cc:1467] Processing CreateTablet for tablet f6f8dfcb870a4461b03415db9c33452e (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.808842 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f8dfcb870a4461b03415db9c33452e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.811957 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:28.812362 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.824287 11309 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:28.825064 11310 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:28.827088 11310 tablet_bootstrap.cc:654] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.829938 11309 tablet_bootstrap.cc:654] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.831210 11302 tablet_copy_client.cc:308] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:28.832754 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 01e27516acca49459176362caf73595e from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:46691)
I20240503 14:13:28.832911 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 01e27516acca49459176362caf73595e from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:46691: session id = 09723f236d8044538d920a37b5d9aa8c-01e27516acca49459176362caf73595e
I20240503 14:13:28.834352 11310 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:28.834537 11310 ts_tablet_manager.cc:1367] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240503 14:13:28.835572 11310 raft_consensus.cc:348] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:28.838124 11310 raft_consensus.cc:374] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.838466 11310 raft_consensus.cc:729] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.838959 11310 consensus_queue.cc:260] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:28.840201 6929 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.840481 11310 ts_tablet_manager.cc:1398] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 14:13:28.840621 11306 tablet_replica.cc:330] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:28.840816 11306 raft_consensus.cc:2229] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:28.841037 11306 raft_consensus.cc:2258] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:28.841275 9819 tablet_copy_source_session.cc:205] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:28.841451 10201 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.841578 11308 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:28.843209 11302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e27516acca49459176362caf73595e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:28.845613 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:28.846848 7060 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:28.850136 10201 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:28.850586 11306 ts_tablet_manager.cc:1875] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:28.852208 11308 tablet_bootstrap.cc:654] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.854461 11302 tablet_copy_client.cc:783] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
W20240503 14:13:28.860090 7127 consensus_peers.cc:485] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d3ec4f0833d4d489f774757312efb32. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:28.860862 11254 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:28.862484 11302 tablet_copy_client.cc:647] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:28.862888 11254 tablet_bootstrap.cc:654] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.867498 11302 tablet_copy_client.cc:523] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:28.871619 11302 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:28.875628 9665 raft_consensus.cc:1261] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:28.876830 11308 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:28.877068 11308 ts_tablet_manager.cc:1367] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.009s
I20240503 14:13:28.877888 11308 raft_consensus.cc:348] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:28.877944 11288 consensus_queue.cc:1035] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.878232 11308 raft_consensus.cc:374] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.878376 11308 raft_consensus.cc:729] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.878373 11306 ts_tablet_manager.cc:1888] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:13:28.878682 11308 consensus_queue.cc:260] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:28.879024 11306 log.cc:1196] T e431272976034209b2c40cb0e460e17a P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/e431272976034209b2c40cb0e460e17a
I20240503 14:13:28.881418 11308 ts_tablet_manager.cc:1398] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:28.883327 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet e431272976034209b2c40cb0e460e17a (table rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]) successfully deleted
I20240503 14:13:28.900158 11302 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:28.900813 11302 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:28.901168 11302 ts_tablet_manager.cc:1367] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.030s user 0.012s sys 0.012s
I20240503 14:13:28.902055 11302 raft_consensus.cc:348] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:28.902396 11302 raft_consensus.cc:729] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:28.902702 11302 consensus_queue.cc:260] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:28.904577 11292 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:28.905000 11292 tablet_bootstrap.cc:492] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:28.905306 11292 ts_tablet_manager.cc:1367] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.139s user 0.055s sys 0.005s
I20240503 14:13:28.906188 11292 raft_consensus.cc:348] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:28.906445 11292 raft_consensus.cc:729] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:28.906657 11292 consensus_queue.cc:260] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:28.907236 11302 ts_tablet_manager.cc:1398] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:28.910480 11292 ts_tablet_manager.cc:1398] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:28.911223 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-01e27516acca49459176362caf73595e received from {username='slave'} at 127.0.186.3:46691
I20240503 14:13:28.911398 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-01e27516acca49459176362caf73595e on tablet 01e27516acca49459176362caf73595e with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:28.914325 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-847ef240dcc64fb794ffcc06e483dc08 received from {username='slave'} at 127.0.186.7:36385
I20240503 14:13:28.914496 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-847ef240dcc64fb794ffcc06e483dc08 on tablet 847ef240dcc64fb794ffcc06e483dc08 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:28.920629 9929 raft_consensus.cc:1261] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:28.921977 11288 consensus_queue.cc:1035] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:28.925845 11255 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:28.928133 11255 tablet_bootstrap.cc:654] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.938290 11255 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:28.938494 11255 ts_tablet_manager.cc:1367] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.002s
I20240503 14:13:28.939306 11255 raft_consensus.cc:348] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:28.941041 11255 raft_consensus.cc:374] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.941165 11255 raft_consensus.cc:729] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.941565 11255 consensus_queue.cc:260] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:28.942121 11322 ts_tablet_manager.cc:902] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:28.943580 11255 ts_tablet_manager.cc:1398] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 14:13:28.943925 11255 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:28.946493 11255 tablet_bootstrap.cc:654] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:28.946965 6655 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 48198af7614b48a2afaa3bb95e071ed3 (table rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c]). Expected version 1 got 0
I20240503 14:13:28.955824 6657 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 48198af7614b48a2afaa3bb95e071ed3 (table rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c]). Expected version 1 got 0
I20240503 14:13:28.958678 11254 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:28.961601 11254 ts_tablet_manager.cc:1367] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.101s user 0.007s sys 0.005s
I20240503 14:13:28.964326 11254 raft_consensus.cc:348] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:28.964519 10201 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.966661 11254 raft_consensus.cc:374] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.966782 11254 raft_consensus.cc:729] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:28.967087 11254 consensus_queue.cc:260] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:28.968183 11326 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:28.968948 11326 tablet.cc:1620] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:28.968658 7060 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.971917 11255 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:28.972806 6929 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:28.969674 11329 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:28.973883 11329 tablet.cc:1620] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:28.974687 11254 ts_tablet_manager.cc:1398] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:28.974959 11254 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:28.977511 11238 ts_tablet_manager.cc:902] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:28.977835 6657 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 48198af7614b48a2afaa3bb95e071ed3 (table rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c]). Expected version 1 got 0
I20240503 14:13:28.978125 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c] alter complete (version 1)
I20240503 14:13:28.990355 6650 catalog_manager.cc:5484] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:28.993829 6654 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 48198af7614b48a2afaa3bb95e071ed3 (table rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c]). Expected version 1 got 0
I20240503 14:13:28.993953 11255 ts_tablet_manager.cc:1367] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.001s
I20240503 14:13:28.994822 11255 raft_consensus.cc:348] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:28.997334 11255 raft_consensus.cc:374] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:28.990586 11328 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:28.999878 11328 tablet.cc:1620] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:29.003834 11322 tablet_copy_client.cc:308] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:29.004956 11255 raft_consensus.cc:729] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.005759 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 6e0bb82edac2464b9d618c1c80807fb2 from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:48891)
I20240503 14:13:29.005932 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 6e0bb82edac2464b9d618c1c80807fb2 from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:48891: session id = f03dc37fe3234033bc665bed4f1b51f5-6e0bb82edac2464b9d618c1c80807fb2
I20240503 14:13:29.006260 11255 consensus_queue.cc:260] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.007063 11255 ts_tablet_manager.cc:1398] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240503 14:13:29.008874 7081 tablet_copy_source_session.cc:205] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:29.010784 11322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0bb82edac2464b9d618c1c80807fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.020479 11309 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:29.020651 11309 ts_tablet_manager.cc:1367] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.196s user 0.003s sys 0.027s
I20240503 14:13:29.021737 11254 tablet_bootstrap.cc:654] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.022593 11309 raft_consensus.cc:348] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.022825 11309 raft_consensus.cc:374] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.022935 11309 raft_consensus.cc:729] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.023380 11322 tablet_copy_client.cc:783] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:29.023279 11309 consensus_queue.cc:260] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.023777 11322 tablet_copy_client.cc:647] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:29.023990 11309 ts_tablet_manager.cc:1398] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:29.024319 11309 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:29.025955 11309 tablet_bootstrap.cc:654] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.027670 11322 tablet_copy_client.cc:523] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:29.029865 11292 ts_tablet_manager.cc:902] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:29.031486 11322 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:29.039462 11292 tablet_copy_client.cc:308] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:29.040289 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 1c727d4554464442a6a81a8ad4996c84 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:58905)
I20240503 14:13:29.040488 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 1c727d4554464442a6a81a8ad4996c84 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:58905: session id = 3efdbd3506274dba846ae40404f0572c-1c727d4554464442a6a81a8ad4996c84
I20240503 14:13:29.042349 11309 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:29.042539 11309 ts_tablet_manager.cc:1367] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.008s
I20240503 14:13:29.043480 11309 raft_consensus.cc:348] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.043771 11309 raft_consensus.cc:374] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.043862 11309 raft_consensus.cc:729] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.043555 6951 tablet_copy_source_session.cc:205] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:29.044131 11309 consensus_queue.cc:260] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.044744 11309 ts_tablet_manager.cc:1398] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:29.052342 11292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c727d4554464442a6a81a8ad4996c84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.060066 11342 ts_tablet_manager.cc:902] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:29.064229 11254 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:29.064496 11254 ts_tablet_manager.cc:1367] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20240503 14:13:29.065564 11254 raft_consensus.cc:348] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.065866 11254 raft_consensus.cc:374] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.065986 11254 raft_consensus.cc:729] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.072250 11254 consensus_queue.cc:260] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.076014 11254 ts_tablet_manager.cc:1398] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20240503 14:13:29.076628 11254 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:29.077011 11292 tablet_copy_client.cc:783] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:29.077894 11292 tablet_copy_client.cc:647] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:29.080127 11254 tablet_bootstrap.cc:654] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.081884 11342 tablet_copy_client.cc:308] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:29.082633 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet a90c8e47862945f1a873b17da14bb397 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:58913)
I20240503 14:13:29.082805 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet a90c8e47862945f1a873b17da14bb397 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:58913: session id = 3efdbd3506274dba846ae40404f0572c-a90c8e47862945f1a873b17da14bb397
I20240503 14:13:29.082974 11121 raft_consensus.cc:482] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.083153 11121 raft_consensus.cc:504] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.083863 11121 leader_election.cc:288] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:29.084954 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7281ccb4c44c8d8cebf78a1af6dd66" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:29.085242 9797 raft_consensus.cc:2454] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:29.085734 7081 tablet_copy_source_session.cc:205] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:29.085740 9864 leader_election.cc:302] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:29.086102 11121 raft_consensus.cc:2781] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.086207 11121 raft_consensus.cc:482] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.086032 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7281ccb4c44c8d8cebf78a1af6dd66" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:29.086323 11121 raft_consensus.cc:3037] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:29.086998 9863 leader_election.cc:341] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:29.087116 11342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a90c8e47862945f1a873b17da14bb397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.089277 11121 raft_consensus.cc:504] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.089944 11121 leader_election.cc:288] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:29.091703 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7281ccb4c44c8d8cebf78a1af6dd66" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:29.091924 9798 raft_consensus.cc:3037] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.093732 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:29.094554 11342 tablet_copy_client.cc:783] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:29.094671 9798 raft_consensus.cc:2454] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:29.094803 11342 tablet_copy_client.cc:647] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:29.096834 11342 tablet_copy_client.cc:523] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:29.097473 9864 leader_election.cc:302] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:29.102203 11342 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:29.084957 11292 tablet_copy_client.cc:523] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:29.108647 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7281ccb4c44c8d8cebf78a1af6dd66" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:29.109798 11328 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
W20240503 14:13:29.110028 9863 leader_election.cc:341] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:29.110428 11328 tablet.cc:1620] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:29.110769 11329 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:29.111425 11329 tablet.cc:1620] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:29.113282 11292 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:29.117493 10933 raft_consensus.cc:2781] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.117686 10933 raft_consensus.cc:686] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:29.117964 10933 consensus_queue.cc:237] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.119400 11118 raft_consensus.cc:482] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.119606 11118 raft_consensus.cc:504] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.120209 11118 leader_election.cc:288] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:29.121630 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=f4d994c819994da78659a9f77b944d3c] alter complete (version 2)
I20240503 14:13:29.124066 6930 consensus_queue.cc:237] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:29.125844 6654 catalog_manager.cc:5484] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:29.126423 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f8dfcb870a4461b03415db9c33452e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:29.126788 6800 raft_consensus.cc:2454] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:29.127625 9863 leader_election.cc:302] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:29.128006 10933 raft_consensus.cc:2781] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.128105 10933 raft_consensus.cc:482] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.128207 10933 raft_consensus.cc:3037] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.131143 10933 raft_consensus.cc:504] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.132530 10933 leader_election.cc:288] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:29.132475 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f8dfcb870a4461b03415db9c33452e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:29.132794 9665 raft_consensus.cc:2454] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:29.133419 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f8dfcb870a4461b03415db9c33452e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:29.133656 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6f8dfcb870a4461b03415db9c33452e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:29.133716 6800 raft_consensus.cc:3037] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.133845 9665 raft_consensus.cc:3037] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.136665 9665 raft_consensus.cc:2454] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:29.137071 9863 leader_election.cc:302] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:29.137418 10933 raft_consensus.cc:2781] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.137668 10933 raft_consensus.cc:686] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:29.137995 10933 consensus_queue.cc:237] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.139700 6800 raft_consensus.cc:2454] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:29.140290 11326 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:29.140972 11326 tablet.cc:1620] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:29.144865 11354 ts_tablet_manager.cc:902] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:29.156067 6654 catalog_manager.cc:5484] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:29.161805 7060 raft_consensus.cc:1203] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:29.176549 11322 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:29.177093 11322 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:29.177413 11322 ts_tablet_manager.cc:1367] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.146s user 0.052s sys 0.000s
I20240503 14:13:29.185644 11254 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:29.185839 11254 ts_tablet_manager.cc:1367] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.000s
I20240503 14:13:29.186575 11254 raft_consensus.cc:348] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.186800 11254 raft_consensus.cc:374] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.186868 11254 raft_consensus.cc:729] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.187166 11254 consensus_queue.cc:260] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
W20240503 14:13:29.189481 6865 consensus_peers.cc:485] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f56e8ea68eb94475838da245b5f060cc. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:29.190272 11254 ts_tablet_manager.cc:1398] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:29.190857 7060 raft_consensus.cc:1261] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:29.178264 11322 raft_consensus.cc:348] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:29.191357 11322 raft_consensus.cc:729] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:29.191570 11322 consensus_queue.cc:260] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:29.191797 10729 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:29.193240 11322 ts_tablet_manager.cc:1398] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240503 14:13:29.193887 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-6e0bb82edac2464b9d618c1c80807fb2 received from {username='slave'} at 127.0.186.6:48891
I20240503 14:13:29.194036 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-6e0bb82edac2464b9d618c1c80807fb2 on tablet 6e0bb82edac2464b9d618c1c80807fb2 with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:29.201036 7191 raft_consensus.cc:1261] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:29.202059 10729 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:29.211777 10729 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:29.228267 7191 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:29.233027 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:29.239213 6658 catalog_manager.cc:5484] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:29.245049 7060 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:29.251709 11132 raft_consensus.cc:482] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.251950 11132 raft_consensus.cc:504] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.252867 11132 leader_election.cc:288] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:29.253628 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaca37078ac4c4eb7f0bb6df0654860" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:29.253942 9797 raft_consensus.cc:2454] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:29.254985 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaca37078ac4c4eb7f0bb6df0654860" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:29.255231 9664 raft_consensus.cc:2454] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:29.257200 7126 leader_election.cc:302] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:29.262079 11328 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:29.265259 10201 raft_consensus.cc:2781] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.265379 10201 raft_consensus.cc:482] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.265493 10201 raft_consensus.cc:3037] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.269521 10201 raft_consensus.cc:504] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.272459 11329 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:29.278235 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaca37078ac4c4eb7f0bb6df0654860" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:29.278510 9797 raft_consensus.cc:3037] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.281313 9797 raft_consensus.cc:2454] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:29.283747 7126 leader_election.cc:302] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:29.284202 10988 raft_consensus.cc:2781] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.284337 10988 raft_consensus.cc:686] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:29.284616 10988 consensus_queue.cc:237] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.286854 9798 raft_consensus.cc:1203] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:29.288686 11326 tablet.cc:1722] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:29.299221 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aaca37078ac4c4eb7f0bb6df0654860" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:29.299638 9664 raft_consensus.cc:3037] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.302549 9664 raft_consensus.cc:2454] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:29.307075 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=f4d994c819994da78659a9f77b944d3c] alter complete (version 3)
I20240503 14:13:29.308991 6655 catalog_manager.cc:5484] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:29.298631 10201 leader_election.cc:288] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:29.315560 9664 raft_consensus.cc:1203] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:29.317598 11238 tablet_copy_client.cc:308] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:29.318969 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 9027b8bfee0443aab7420df3f7257bcb from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:46745)
I20240503 14:13:29.319142 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 9027b8bfee0443aab7420df3f7257bcb from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:46745: session id = 5b7e36bf354f480c90859cb8867b59cf-9027b8bfee0443aab7420df3f7257bcb
I20240503 14:13:29.322250 6951 tablet_copy_source_session.cc:205] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:29.325491 11238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9027b8bfee0443aab7420df3f7257bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.331012 11354 tablet_copy_client.cc:308] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:29.334905 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet a44ed92d65114bfda2e0f38054b17ada from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:48891)
I20240503 14:13:29.335119 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet a44ed92d65114bfda2e0f38054b17ada from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:48891: session id = f03dc37fe3234033bc665bed4f1b51f5-a44ed92d65114bfda2e0f38054b17ada
I20240503 14:13:29.338215 7081 tablet_copy_source_session.cc:205] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:29.340157 11238 tablet_copy_client.cc:783] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:29.340557 11238 tablet_copy_client.cc:647] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:29.342303 11238 tablet_copy_client.cc:523] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:29.344954 6930 consensus_queue.cc:237] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:29.345187 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44ed92d65114bfda2e0f38054b17ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.345691 11238 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
W20240503 14:13:29.358834 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f30640fc81e4aeead38e3f8659c056a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:29.361263 9530 raft_consensus.cc:1261] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:29.361794 7060 raft_consensus.cc:1261] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:29.364158 10729 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:29.365581 9872 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 14:13:29.365833 11062 raft_consensus.cc:1050] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:29.367220 11062 consensus_queue.cc:237] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:29.372514 11222 raft_consensus.cc:482] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.372685 11222 raft_consensus.cc:504] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:29.373456 11222 leader_election.cc:288] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:29.374068 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7281ccb4c44c8d8cebf78a1af6dd66" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:29.374884 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c7281ccb4c44c8d8cebf78a1af6dd66" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:29.375139 9797 raft_consensus.cc:2379] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f03dc37fe3234033bc665bed4f1b51f5 in current term 1: Already voted for candidate ea801601713a4d55aee0de9e3473e0bc in this term.
I20240503 14:13:29.376133 9600 leader_election.cc:302] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5; no voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:29.376631 11222 raft_consensus.cc:3037] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.379458 11222 raft_consensus.cc:2726] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:29.385263 9532 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.385998 10729 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.390241 10729 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:29.390866 11354 tablet_copy_client.cc:783] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:29.391857 11354 tablet_copy_client.cc:647] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:29.393689 11354 tablet_copy_client.cc:523] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:29.394578 9531 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:29.394881 7059 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:29.402189 6654 catalog_manager.cc:5484] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:29.405371 11342 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:29.405805 11342 tablet_bootstrap.cc:492] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:29.406122 11342 ts_tablet_manager.cc:1367] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.304s user 0.088s sys 0.077s
I20240503 14:13:29.406535 7191 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.406939 11342 raft_consensus.cc:348] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:29.407202 11342 raft_consensus.cc:729] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:29.407127 9872 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.407395 11342 consensus_queue.cc:260] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:29.409322 7060 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.411237 9872 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.412113 11342 ts_tablet_manager.cc:1398] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20240503 14:13:29.412784 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-a90c8e47862945f1a873b17da14bb397 received from {username='slave'} at 127.0.186.7:58913
I20240503 14:13:29.412921 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-a90c8e47862945f1a873b17da14bb397 on tablet a90c8e47862945f1a873b17da14bb397 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:29.414209 6655 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:29.414419 6655 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:29.417914 6655 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 48198af7614b48a2afaa3bb95e071ed3
I20240503 14:13:29.419502 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 48198af7614b48a2afaa3bb95e071ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:29.420090 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 48198af7614b48a2afaa3bb95e071ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:29.422118 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 48198af7614b48a2afaa3bb95e071ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:29 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:29.436147 9872 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.438714 11238 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:29.439172 11238 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:29.439450 11238 ts_tablet_manager.cc:1367] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.094s user 0.045s sys 0.003s
I20240503 14:13:29.440311 11238 raft_consensus.cc:348] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:29.440587 11238 raft_consensus.cc:729] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:29.440825 11238 consensus_queue.cc:260] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:29.442870 7060 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.443636 9797 raft_consensus.cc:1203] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:29.445058 11238 ts_tablet_manager.cc:1398] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 14:13:29.448493 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-9027b8bfee0443aab7420df3f7257bcb received from {username='slave'} at 127.0.186.5:46745
I20240503 14:13:29.448655 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-9027b8bfee0443aab7420df3f7257bcb on tablet 9027b8bfee0443aab7420df3f7257bcb with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:29.450459 11238 ts_tablet_manager.cc:902] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:29.473635 6655 catalog_manager.cc:5484] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.483165 11354 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:29.488004 11386 tablet_replica.cc:330] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:29.488323 11386 raft_consensus.cc:2229] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:29.488916 11386 raft_consensus.cc:2258] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.491775 9532 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.493916 9531 raft_consensus.cc:1203] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:29.509724 11386 ts_tablet_manager.cc:1875] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.512059 11384 tablet_replica.cc:330] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:29.512369 11384 raft_consensus.cc:2229] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.512670 11384 raft_consensus.cc:2258] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.512691 11385 tablet_replica.cc:330] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:29.512857 11385 raft_consensus.cc:2229] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.513020 11385 raft_consensus.cc:2258] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.514006 11385 ts_tablet_manager.cc:1875] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.517513 7191 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.518134 11384 ts_tablet_manager.cc:1875] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:29.519419 11386 ts_tablet_manager.cc:1888] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:29.519989 6930 consensus_queue.cc:237] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:29.520938 11238 tablet_copy_client.cc:308] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:29.522686 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:29.527693 9532 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:29.529994 11388 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 14:13:29.531459 11385 ts_tablet_manager.cc:1888] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:29.531659 11385 log.cc:1196] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/48198af7614b48a2afaa3bb95e071ed3
I20240503 14:13:29.532160 11385 ts_tablet_manager.cc:1909] T 48198af7614b48a2afaa3bb95e071ed3 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:29.533339 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 48198af7614b48a2afaa3bb95e071ed3 (table rebalancer_dynamic_table_new_name [id=f4d994c819994da78659a9f77b944d3c]) successfully deleted
I20240503 14:13:29.535490 7060 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:29.537333 11386 log.cc:1196] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/48198af7614b48a2afaa3bb95e071ed3
I20240503 14:13:29.537827 11386 ts_tablet_manager.cc:1909] T 48198af7614b48a2afaa3bb95e071ed3 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:29.538621 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 413becdc566741348329e37bb740e9c1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]), partition=
I20240503 14:13:29.539000 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413becdc566741348329e37bb740e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.542268 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 413becdc566741348329e37bb740e9c1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]), partition=
I20240503 14:13:29.542660 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413becdc566741348329e37bb740e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.543897 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 48198af7614b48a2afaa3bb95e071ed3 (table rebalancer_dynamic_table_new_name [id=f4d994c819994da78659a9f77b944d3c]) successfully deleted
I20240503 14:13:29.548748 10729 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:29.549548 10729 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.555467 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:29.556159 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 413becdc566741348329e37bb740e9c1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]), partition=
I20240503 14:13:29.556581 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 413becdc566741348329e37bb740e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.556774 11384 ts_tablet_manager.cc:1888] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:29.556949 11384 log.cc:1196] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/48198af7614b48a2afaa3bb95e071ed3
I20240503 14:13:29.557408 11384 ts_tablet_manager.cc:1909] T 48198af7614b48a2afaa3bb95e071ed3 P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:29.559774 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 48198af7614b48a2afaa3bb95e071ed3 (table rebalancer_dynamic_table_new_name [id=f4d994c819994da78659a9f77b944d3c]) successfully deleted
I20240503 14:13:29.565555 7060 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.566785 10933 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.569489 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 1d3ec4f0833d4d489f774757312efb32 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:48297)
I20240503 14:13:29.569696 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 1d3ec4f0833d4d489f774757312efb32 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:48297: session id = 5b7e36bf354f480c90859cb8867b59cf-1d3ec4f0833d4d489f774757312efb32
I20240503 14:13:29.573802 6650 catalog_manager.cc:5484] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.575244 7211 tablet_copy_source_session.cc:205] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:29.577574 11238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3ec4f0833d4d489f774757312efb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.580014 9532 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:29.583895 11254 tablet_bootstrap.cc:492] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:29.585696 11254 tablet_bootstrap.cc:654] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.595633 11118 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.601183 11238 tablet_copy_client.cc:783] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:29.601490 11238 tablet_copy_client.cc:647] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:29.602540 11254 tablet_bootstrap.cc:492] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:29.602735 11254 ts_tablet_manager.cc:1367] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240503 14:13:29.605077 11254 raft_consensus.cc:348] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.605382 11254 raft_consensus.cc:374] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.605497 11254 raft_consensus.cc:729] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.605779 11254 consensus_queue.cc:260] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.607039 11254 ts_tablet_manager.cc:1398] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:29.608511 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:29.611420 10698 raft_consensus.cc:1050] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:29.612476 10698 consensus_queue.cc:237] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:29.623471 11238 tablet_copy_client.cc:523] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:29.640201 11408 tablet_replica.cc:330] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:29.640463 11408 raft_consensus.cc:2229] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.640718 11408 raft_consensus.cc:2258] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.643076 11238 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:29.648722 11408 ts_tablet_manager.cc:1875] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:29.653602 9929 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:29.654178 7060 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:29.654487 11414 tablet_bootstrap.cc:492] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:29.655367 11408 ts_tablet_manager.cc:1888] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:29.657109 11225 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:29.657961 11225 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:29.658402 9532 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:29.661809 11292 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:29.662333 11292 tablet_bootstrap.cc:492] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:29.662587 11292 ts_tablet_manager.cc:1367] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.549s user 0.051s sys 0.012s
I20240503 14:13:29.663463 11292 raft_consensus.cc:348] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:29.663718 11292 raft_consensus.cc:729] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:29.663954 11292 consensus_queue.cc:260] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:29.664304 11225 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:29.665241 11408 log.cc:1196] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/b3e2d7664e6c45d69f4f8b20dcf0e4d0
I20240503 14:13:29.666492 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 (table rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]) successfully deleted
I20240503 14:13:29.673687 11132 raft_consensus.cc:1050] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:29.674867 11132 consensus_queue.cc:237] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:29.677716 11401 tablet_bootstrap.cc:492] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:29.681700 11401 tablet_bootstrap.cc:654] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.684033 11292 ts_tablet_manager.cc:1398] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 14:13:29.690881 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-1c727d4554464442a6a81a8ad4996c84 received from {username='slave'} at 127.0.186.7:58905
I20240503 14:13:29.691097 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-1c727d4554464442a6a81a8ad4996c84 on tablet 1c727d4554464442a6a81a8ad4996c84 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:29.697715 11009 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.702399 11401 tablet_bootstrap.cc:492] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:29.702561 11401 ts_tablet_manager.cc:1367] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20240503 14:13:29.703472 11401 raft_consensus.cc:348] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.703706 11401 raft_consensus.cc:374] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:29.703779 11401 raft_consensus.cc:729] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:29.704005 11401 consensus_queue.cc:260] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.705672 11401 ts_tablet_manager.cc:1398] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:29.707988 7060 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.718101 10785 raft_consensus.cc:1050] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:29.718309 9532 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.718967 10785 consensus_queue.cc:237] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:29.719254 11400 raft_consensus.cc:482] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:29.719415 11400 raft_consensus.cc:504] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.720144 11400 leader_election.cc:288] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:29.720813 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413becdc566741348329e37bb740e9c1" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:29.720882 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413becdc566741348329e37bb740e9c1" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:29.721096 9664 raft_consensus.cc:2454] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
W20240503 14:13:29.721796 6736 leader_election.cc:341] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:29.721952 6736 leader_election.cc:302] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13; no voters: 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:29.722424 11400 raft_consensus.cc:2781] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:29.722553 11400 raft_consensus.cc:482] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:29.722646 11400 raft_consensus.cc:3037] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:29.724980 11118 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.725625 11400 raft_consensus.cc:504] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.725507 6654 catalog_manager.cc:5484] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.726748 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413becdc566741348329e37bb740e9c1" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:29.727213 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "413becdc566741348329e37bb740e9c1" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:29.727429 9664 raft_consensus.cc:3037] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:29.727964 6736 leader_election.cc:341] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:29.729879 11400 leader_election.cc:288] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:29.732712 9664 raft_consensus.cc:2454] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:29.733521 6734 leader_election.cc:302] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13; no voters: 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:29.733992 11400 raft_consensus.cc:2781] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:29.734126 11400 raft_consensus.cc:686] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:29.734377 11400 consensus_queue.cc:237] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.736775 11238 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:29.737380 11238 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:29.737725 11238 ts_tablet_manager.cc:1367] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.095s user 0.045s sys 0.010s
I20240503 14:13:29.738551 11238 raft_consensus.cc:348] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:29.738896 11238 raft_consensus.cc:729] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:29.738816 10201 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.739176 11238 consensus_queue.cc:260] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:29.739365 11118 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.743872 11238 ts_tablet_manager.cc:1398] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:29.748400 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-1d3ec4f0833d4d489f774757312efb32 received from {username='slave'} at 127.0.186.5:48297
I20240503 14:13:29.748594 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-1d3ec4f0833d4d489f774757312efb32 on tablet 1d3ec4f0833d4d489f774757312efb32 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:29.772023 11414 tablet_bootstrap.cc:654] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:29.774071 11354 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:29.774577 11354 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:29.775027 11354 ts_tablet_manager.cc:1367] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.292s user 0.055s sys 0.000s
I20240503 14:13:29.775934 11354 raft_consensus.cc:348] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:29.776235 11354 raft_consensus.cc:729] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:29.776449 11354 consensus_queue.cc:260] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:29.777172 11354 ts_tablet_manager.cc:1398] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:29.779721 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-a44ed92d65114bfda2e0f38054b17ada received from {username='slave'} at 127.0.186.6:48891
I20240503 14:13:29.779870 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-a44ed92d65114bfda2e0f38054b17ada on tablet a44ed92d65114bfda2e0f38054b17ada with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:29.780532 6655 catalog_manager.cc:5484] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:29.781730 6799 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.782914 7191 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.784309 10785 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.784713 11040 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.791692 11415 ts_tablet_manager.cc:902] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:29.798302 9664 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.799090 10785 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.805968 11370 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.814972 9796 raft_consensus.cc:1261] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.815783 11370 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.835696 6800 raft_consensus.cc:1261] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.835914 7060 raft_consensus.cc:1261] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:29.836556 10988 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.837131 10988 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:29.837664 9794 consensus_queue.cc:237] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:29.840392 7060 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:29.841014 11422 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:29.844228 11225 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.848294 7060 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.853682 9532 raft_consensus.cc:1203] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:29.859256 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01e27516acca49459176362caf73595e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:29.861590 6654 catalog_manager.cc:5484] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.866993 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e27516acca49459176362caf73595e with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:29.867211 9929 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:29.867396 11386 tablet_replica.cc:330] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:29.867666 11386 raft_consensus.cc:2229] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:29.867906 11386 raft_consensus.cc:2258] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:29.867820 11225 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:29.869246 11386 ts_tablet_manager.cc:1875] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:29.868948 9928 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.871546 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:29.875614 9928 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:29.888254 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 77d3f36de08d4f1986e42e86e1b95fbf (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.888703 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d3f36de08d4f1986e42e86e1b95fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.892906 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.893425 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.896855 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 77d3f36de08d4f1986e42e86e1b95fbf (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.897403 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d3f36de08d4f1986e42e86e1b95fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.902292 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 77d3f36de08d4f1986e42e86e1b95fbf (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.902865 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d3f36de08d4f1986e42e86e1b95fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.903129 6798 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:29.908051 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.908581 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.899089 10785 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:29.922327 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:29.924911 9665 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:29.932881 6911 tablet_service.cc:1467] Processing CreateTablet for tablet e478266c296c473eb7ebd1f5ebf7d5cf (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.933466 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e478266c296c473eb7ebd1f5ebf7d5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.932881 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.933885 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.936161 9910 tablet_service.cc:1467] Processing CreateTablet for tablet e478266c296c473eb7ebd1f5ebf7d5cf (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.936697 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e478266c296c473eb7ebd1f5ebf7d5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.942909 7170 tablet_service.cc:1467] Processing CreateTablet for tablet e478266c296c473eb7ebd1f5ebf7d5cf (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:29.943459 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e478266c296c473eb7ebd1f5ebf7d5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:29.931183 6657 catalog_manager.cc:5484] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:29.966687 11354 ts_tablet_manager.cc:902] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
W20240503 14:13:29.975275 6736 consensus_peers.cc:485] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:29.976915 9663 raft_consensus.cc:1261] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:29.978281 11400 consensus_queue.cc:1035] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:29.985373 7060 consensus_queue.cc:237] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:29.988114 11414 tablet_bootstrap.cc:492] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:29.989694 11414 ts_tablet_manager.cc:1367] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.343s user 0.012s sys 0.040s
I20240503 14:13:29.989790 9664 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:29.991397 10785 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:29.992995 11414 raft_consensus.cc:348] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:29.995497 11386 ts_tablet_manager.cc:1888] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:29.998024 11040 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:30.000720 10988 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.003468 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e0bb82edac2464b9d618c1c80807fb2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:30.005574 9664 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:30.007097 7060 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.011940 6799 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:30.012017 9797 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.013751 11040 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:30.014977 6650 catalog_manager.cc:5484] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:29.995564 11414 raft_consensus.cc:374] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.019872 11414 raft_consensus.cc:729] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.020215 11414 consensus_queue.cc:260] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.020331 7189 consensus_queue.cc:237] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:30.020915 11414 ts_tablet_manager.cc:1398] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240503 14:13:30.021256 11414 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:30.022739 7060 raft_consensus.cc:1261] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:30.023129 11414 tablet_bootstrap.cc:654] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.023335 10988 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:30.019889 11386 log.cc:1196] T 01e27516acca49459176362caf73595e P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/01e27516acca49459176362caf73595e
I20240503 14:13:30.029525 6650 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]). Expected version 1 got 0
I20240503 14:13:30.030664 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 01e27516acca49459176362caf73595e (table rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]) successfully deleted
I20240503 14:13:30.032538 9798 raft_consensus.cc:1261] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:30.033051 9872 raft_consensus.cc:1050] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:30.033030 6655 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]). Expected version 1 got 0
I20240503 14:13:30.034096 10988 consensus_queue.cc:1035] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:30.034072 9872 consensus_queue.cc:237] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:30.037268 9798 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.040287 10988 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.040925 6799 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:30.044435 11443 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:30.047376 9532 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.047421 6655 catalog_manager.cc:5484] T 847ef240dcc64fb794ffcc06e483dc08 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 5 to 6, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.048048 10729 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.048054 11447 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:30.053139 7058 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.053817 7189 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.053887 10729 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.054973 10729 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.011885 6657 catalog_manager.cc:5484] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:30.066701 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 847ef240dcc64fb794ffcc06e483dc08 with delete_type TABLET_DATA_TOMBSTONED (TS f1a669e28f014fe287932acf56bebc13 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:30.066953 10371 raft_consensus.cc:1050] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:30.068295 10371 consensus_queue.cc:237] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:30.070338 11444 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:30.074766 11444 tablet_bootstrap.cc:654] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.080272 11447 tablet_bootstrap.cc:654] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.081544 11443 tablet_bootstrap.cc:654] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.083479 7060 raft_consensus.cc:2932] T 847ef240dcc64fb794ffcc06e483dc08 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.086215 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0bb82edac2464b9d618c1c80807fb2 with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:30.086658 11408 tablet_replica.cc:330] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:30.086916 11408 raft_consensus.cc:2229] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.087100 11408 pending_rounds.cc:62] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: Trying to abort 1 pending ops.
I20240503 14:13:30.087215 11408 pending_rounds.cc:68] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598017406406656 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "6e0bb82edac2464b9d618c1c80807fb2" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } } }
I20240503 14:13:30.087805 11408 raft_consensus.cc:2894] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:13:30.087946 11408 raft_consensus.cc:2866] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:30.088084 11408 raft_consensus.cc:2258] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.090822 11408 ts_tablet_manager.cc:1875] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:30.092072 11449 tablet_bootstrap.cc:492] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:30.094571 11449 tablet_bootstrap.cc:654] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.096160 11448 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:30.098079 11457 tablet_replica.cc:330] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:30.098838 11448 tablet_bootstrap.cc:654] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.100849 11457 raft_consensus.cc:2229] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.101161 11457 pending_rounds.cc:62] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: Trying to abort 1 pending ops.
I20240503 14:13:30.101347 11457 pending_rounds.cc:68] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023598017225240576 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "847ef240dcc64fb794ffcc06e483dc08" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } } }
I20240503 14:13:30.101965 11457 raft_consensus.cc:2894] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240503 14:13:30.102154 11457 raft_consensus.cc:2866] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:30.102344 11457 raft_consensus.cc:2258] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.105288 11444 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:30.105499 11444 ts_tablet_manager.cc:1367] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.007s
I20240503 14:13:30.105497 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 847ef240dcc64fb794ffcc06e483dc08 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:30.106429 11444 raft_consensus.cc:348] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.106673 11444 raft_consensus.cc:374] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.106786 11444 raft_consensus.cc:729] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.107041 11444 consensus_queue.cc:260] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.107661 11444 ts_tablet_manager.cc:1398] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:30.111886 11354 tablet_copy_client.cc:308] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:30.112782 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 9f30640fc81e4aeead38e3f8659c056a from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:60021)
I20240503 14:13:30.112993 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 9f30640fc81e4aeead38e3f8659c056a from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:60021: session id = f03dc37fe3234033bc665bed4f1b51f5-9f30640fc81e4aeead38e3f8659c056a
I20240503 14:13:30.114809 9797 raft_consensus.cc:1203] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:30.113850 11457 ts_tablet_manager.cc:1875] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:30.116003 6951 tablet_copy_source_session.cc:205] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:30.121876 11354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f30640fc81e4aeead38e3f8659c056a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.122352 11457 ts_tablet_manager.cc:1888] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:30.122628 6650 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]). Expected version 1 got 0
I20240503 14:13:30.126824 11408 ts_tablet_manager.cc:1888] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:30.126986 11414 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:30.127218 11414 ts_tablet_manager.cc:1367] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.106s user 0.014s sys 0.013s
I20240503 14:13:30.127346 11444 tablet_bootstrap.cc:492] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:30.128257 11414 raft_consensus.cc:348] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.128513 11414 raft_consensus.cc:374] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.128610 11414 raft_consensus.cc:729] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.128952 11414 consensus_queue.cc:260] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.129779 11408 log.cc:1196] T 6e0bb82edac2464b9d618c1c80807fb2 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/6e0bb82edac2464b9d618c1c80807fb2
I20240503 14:13:30.130954 11444 tablet_bootstrap.cc:654] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.131968 11414 ts_tablet_manager.cc:1398] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 14:13:30.132488 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 6e0bb82edac2464b9d618c1c80807fb2 (table rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]) successfully deleted
I20240503 14:13:30.143472 11444 tablet_bootstrap.cc:492] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:30.143795 11444 ts_tablet_manager.cc:1367] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240503 14:13:30.145053 11444 raft_consensus.cc:348] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.145404 11444 raft_consensus.cc:374] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.145548 11444 raft_consensus.cc:729] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.145912 11444 consensus_queue.cc:260] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.147017 11448 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:30.147174 11448 ts_tablet_manager.cc:1367] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20240503 14:13:30.147961 11444 ts_tablet_manager.cc:1398] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:30.148032 6650 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]). Expected version 1 got 0
I20240503 14:13:30.148092 11448 raft_consensus.cc:348] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.149111 11448 raft_consensus.cc:374] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.151420 11449 tablet_bootstrap.cc:492] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:30.152087 11449 ts_tablet_manager.cc:1367] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.003s
I20240503 14:13:30.153080 11449 raft_consensus.cc:348] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.153201 11457 log.cc:1196] T 847ef240dcc64fb794ffcc06e483dc08 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/847ef240dcc64fb794ffcc06e483dc08
I20240503 14:13:30.153276 11448 raft_consensus.cc:729] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.153887 11449 raft_consensus.cc:374] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.154114 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 847ef240dcc64fb794ffcc06e483dc08 (table rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]) successfully deleted
I20240503 14:13:30.120450 11464 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:30.154093 11463 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:30.154629 11463 tablet.cc:1620] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:30.154633 11464 tablet.cc:1620] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:30.155975 11443 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:30.156201 11443 ts_tablet_manager.cc:1367] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.002s
I20240503 14:13:30.157042 11443 raft_consensus.cc:348] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.158749 6930 raft_consensus.cc:1261] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.159631 11449 raft_consensus.cc:729] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.160069 11447 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:30.160118 9797 raft_consensus.cc:1261] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.160791 11449 consensus_queue.cc:260] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.161253 6800 raft_consensus.cc:1261] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.161993 11388 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.165668 11447 ts_tablet_manager.cc:1367] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.000s
I20240503 14:13:30.153896 11448 consensus_queue.cc:260] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.159376 10785 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240503 14:13:30.168252 10785 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.168764 11040 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.169433 9532 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.166213 7060 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.175560 6650 catalog_manager.cc:5484] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:30.181456 11449 ts_tablet_manager.cc:1398] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
I20240503 14:13:30.183387 7189 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.187995 11443 raft_consensus.cc:374] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.188252 11443 raft_consensus.cc:729] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.188658 11443 consensus_queue.cc:260] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.189741 11443 ts_tablet_manager.cc:1398] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
I20240503 14:13:30.166637 11447 raft_consensus.cc:348] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.191556 11447 raft_consensus.cc:374] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.192713 11447 raft_consensus.cc:729] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.194191 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003] alter complete (version 1)
I20240503 14:13:30.193070 11447 consensus_queue.cc:260] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.195235 11447 ts_tablet_manager.cc:1398] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.029s user 0.000s sys 0.002s
I20240503 14:13:30.199735 11447 tablet_bootstrap.cc:492] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:30.199815 11354 tablet_copy_client.cc:783] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:30.200193 11354 tablet_copy_client.cc:647] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:30.207515 11040 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.208385 11447 tablet_bootstrap.cc:654] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.214455 6930 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.215003 6929 consensus_queue.cc:237] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:30.219329 9532 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:30.219739 7191 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:30.220160 11388 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:30.221905 10729 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:30.226259 9530 raft_consensus.cc:3037] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.228588 11448 ts_tablet_manager.cc:1398] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.081s user 0.000s sys 0.003s
I20240503 14:13:30.228928 11448 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:30.231391 6800 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.234843 10729 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.237365 9663 raft_consensus.cc:1203] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:30.244941 11448 tablet_bootstrap.cc:654] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.246703 11195 raft_consensus.cc:482] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.246937 11195 raft_consensus.cc:504] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.247849 11195 leader_election.cc:288] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:30.249156 9532 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.252282 7190 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.254537 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9027b8bfee0443aab7420df3f7257bcb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:30.255517 10988 raft_consensus.cc:482] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.255753 10988 raft_consensus.cc:504] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.256523 10988 leader_election.cc:288] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:30.259887 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e478266c296c473eb7ebd1f5ebf7d5cf" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:30.267032 6655 catalog_manager.cc:5484] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.269230 10988 raft_consensus.cc:1050] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:30.270391 10988 consensus_queue.cc:237] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:30.266633 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e478266c296c473eb7ebd1f5ebf7d5cf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:30.271317 7189 raft_consensus.cc:2454] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:30.277702 10933 raft_consensus.cc:482] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.277882 10933 raft_consensus.cc:504] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.277595 6653 catalog_manager.cc:5484] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.278626 10933 leader_election.cc:288] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:30.279233 9532 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.275955 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e478266c296c473eb7ebd1f5ebf7d5cf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
W20240503 14:13:30.280136 9863 leader_election.cc:341] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:30.281004 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:30.281216 9532 raft_consensus.cc:2454] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:30.281693 9865 leader_election.cc:302] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:30.281844 7060 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.281970 10933 raft_consensus.cc:2781] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.282052 10933 raft_consensus.cc:482] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.282148 10933 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.282553 10201 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.283142 10988 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.285015 10933 raft_consensus.cc:504] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.286273 10933 leader_election.cc:288] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:30.286994 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:30.287227 9533 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.290109 9533 raft_consensus.cc:2454] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:30.290935 9865 leader_election.cc:302] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:30.291301 11195 raft_consensus.cc:2781] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.291424 11195 raft_consensus.cc:686] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:30.291782 11195 consensus_queue.cc:237] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.295253 11493 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:30.295871 11493 tablet.cc:1620] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:30.295915 11447 tablet_bootstrap.cc:492] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:30.296073 11447 ts_tablet_manager.cc:1367] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.003s
I20240503 14:13:30.296954 11447 raft_consensus.cc:348] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.297334 11447 raft_consensus.cc:374] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.297472 11447 raft_consensus.cc:729] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.297755 11447 consensus_queue.cc:260] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
W20240503 14:13:30.301617 7125 leader_election.cc:341] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:30.302678 11370 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.303033 9531 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.307528 7060 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.309139 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 9027b8bfee0443aab7420df3f7257bcb with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:30.314615 6653 catalog_manager.cc:5484] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:30.317668 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:30.299849 6929 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.320755 10201 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.327210 6653 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]). Expected version 2 got 0
I20240503 14:13:30.331282 6653 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]). Expected version 2 got 1
I20240503 14:13:30.336261 6655 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]). Expected version 2 got 1
I20240503 14:13:30.337594 11464 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:30.338227 11464 tablet.cc:1620] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:30.338420 11463 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:30.338739 11447 ts_tablet_manager.cc:1398] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240503 14:13:30.338922 11463 tablet.cc:1620] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:30.339850 11415 tablet_copy_client.cc:308] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:30.247753 9796 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.343691 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet f56e8ea68eb94475838da245b5f060cc from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:39445)
I20240503 14:13:30.343850 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet f56e8ea68eb94475838da245b5f060cc from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:39445: session id = ea801601713a4d55aee0de9e3473e0bc-f56e8ea68eb94475838da245b5f060cc
I20240503 14:13:30.346701 6951 tablet_copy_source_session.cc:205] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:30.347754 9795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:30.316103 11009 raft_consensus.cc:482] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.348404 11009 raft_consensus.cc:504] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:30.349126 11009 leader_election.cc:288] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:30.349782 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:30.347488 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:30.351423 11415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f56e8ea68eb94475838da245b5f060cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.351457 9864 leader_election.cc:302] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, ea801601713a4d55aee0de9e3473e0bc; no voters: 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:30.351487 9795 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.351776 10933 raft_consensus.cc:2781] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.351892 10933 raft_consensus.cc:482] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.351982 10933 raft_consensus.cc:3037] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.354415 9795 raft_consensus.cc:2454] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:30.354784 10933 raft_consensus.cc:504] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.356140 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e478266c296c473eb7ebd1f5ebf7d5cf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:30.356395 6931 raft_consensus.cc:3037] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.356433 10933 leader_election.cc:288] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:30.359021 6931 raft_consensus.cc:2454] T e478266c296c473eb7ebd1f5ebf7d5cf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:30.358978 6653 catalog_manager.cc:5484] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:30.360272 9863 leader_election.cc:302] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:30.363655 10933 raft_consensus.cc:2781] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.364185 10933 raft_consensus.cc:686] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:30.364562 10933 consensus_queue.cc:237] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:30.364871 6653 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003]). Expected version 2 got 1
I20240503 14:13:30.366081 11415 tablet_copy_client.cc:783] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:30.366356 11415 tablet_copy_client.cc:647] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:30.368126 7189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e478266c296c473eb7ebd1f5ebf7d5cf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:30.368247 11415 tablet_copy_client.cc:523] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:30.368463 7189 raft_consensus.cc:3037] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.369244 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:30.370090 9531 raft_consensus.cc:2379] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3efdbd3506274dba846ae40404f0572c in current term 1: Already voted for candidate ea801601713a4d55aee0de9e3473e0bc in this term.
I20240503 14:13:30.370645 9735 leader_election.cc:302] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:30.371223 11009 raft_consensus.cc:2726] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:30.371273 7189 raft_consensus.cc:2454] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:30.371678 11493 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:30.372282 11493 tablet.cc:1620] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:30.372632 11415 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:30.339325 11494 tablet_replica.cc:330] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:30.351114 7060 consensus_queue.cc:237] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:30.380000 11494 raft_consensus.cc:2229] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.380555 6655 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=6141d88e5e93405cabc65badf2126003] alter complete (version 2)
I20240503 14:13:30.381016 11494 raft_consensus.cc:2258] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.380671 11354 tablet_copy_client.cc:523] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:30.383317 9796 raft_consensus.cc:1261] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:30.383358 6800 raft_consensus.cc:1261] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:30.383944 11040 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:30.386155 10371 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:30.388860 10371 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.393793 6800 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.402693 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a90c8e47862945f1a873b17da14bb397 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:30.407439 9796 raft_consensus.cc:2932] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.411703 6653 catalog_manager.cc:5484] T a90c8e47862945f1a873b17da14bb397 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:30.415833 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet a90c8e47862945f1a873b17da14bb397 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:30.429347 11448 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:30.429986 6650 catalog_manager.cc:5484] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:30.432801 11494 ts_tablet_manager.cc:1875] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:30.433137 11448 ts_tablet_manager.cc:1367] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.204s user 0.001s sys 0.003s
I20240503 14:13:30.436349 11415 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:30.436816 11415 tablet_bootstrap.cc:492] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:30.437117 11415 ts_tablet_manager.cc:1367] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.065s user 0.043s sys 0.011s
I20240503 14:13:30.437994 11415 raft_consensus.cc:348] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:30.438262 11415 raft_consensus.cc:729] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:30.438491 11415 consensus_queue.cc:260] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:30.439070 11415 ts_tablet_manager.cc:1398] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:30.441073 11494 ts_tablet_manager.cc:1888] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:30.441195 11494 log.cc:1196] T 9027b8bfee0443aab7420df3f7257bcb P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/9027b8bfee0443aab7420df3f7257bcb
I20240503 14:13:30.441510 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-f56e8ea68eb94475838da245b5f060cc received from {username='slave'} at 127.0.186.8:39445
I20240503 14:13:30.441670 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-f56e8ea68eb94475838da245b5f060cc on tablet f56e8ea68eb94475838da245b5f060cc with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:30.441447 6929 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:30.443167 11500 tablet_replica.cc:330] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:30.443392 11500 raft_consensus.cc:2229] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.442831 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 9027b8bfee0443aab7420df3f7257bcb (table rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]) successfully deleted
I20240503 14:13:30.443774 11500 raft_consensus.cc:2258] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.447193 11500 ts_tablet_manager.cc:1875] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:30.447795 11448 raft_consensus.cc:348] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.448037 11448 raft_consensus.cc:374] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.448113 11448 raft_consensus.cc:729] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.448354 11448 consensus_queue.cc:260] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.448937 11448 ts_tablet_manager.cc:1398] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240503 14:13:30.450457 11354 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:30.453831 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e478266c296c473eb7ebd1f5ebf7d5cf" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:30.454488 7125 leader_election.cc:302] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:30.454785 10201 raft_consensus.cc:2726] T e478266c296c473eb7ebd1f5ebf7d5cf P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:30.470438 11500 ts_tablet_manager.cc:1888] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:30.471187 9872 raft_consensus.cc:482] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.475052 11500 log.cc:1196] T a90c8e47862945f1a873b17da14bb397 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/a90c8e47862945f1a873b17da14bb397
I20240503 14:13:30.475129 9872 raft_consensus.cc:504] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.476210 9872 leader_election.cc:288] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:30.477237 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet a90c8e47862945f1a873b17da14bb397 (table rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]) successfully deleted
I20240503 14:13:30.479537 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ddfa17abc4ad2a64d8015724b4ae4" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:30.479856 9796 raft_consensus.cc:2454] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:30.480288 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ddfa17abc4ad2a64d8015724b4ae4" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:30.480602 7060 raft_consensus.cc:2454] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:30.481750 6867 leader_election.cc:302] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:30.482329 11388 raft_consensus.cc:2781] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.482442 11388 raft_consensus.cc:482] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.482537 11388 raft_consensus.cc:3037] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.483091 11354 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:30.483489 11354 tablet_bootstrap.cc:492] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:30.483840 11354 ts_tablet_manager.cc:1367] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20240503 14:13:30.484990 11388 raft_consensus.cc:504] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.484892 11354 raft_consensus.cc:348] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:30.485208 11354 raft_consensus.cc:729] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:30.485468 11354 consensus_queue.cc:260] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:30.486255 11354 ts_tablet_manager.cc:1398] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:30.487088 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-9f30640fc81e4aeead38e3f8659c056a received from {username='slave'} at 127.0.186.6:60021
I20240503 14:13:30.490656 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-9f30640fc81e4aeead38e3f8659c056a on tablet 9f30640fc81e4aeead38e3f8659c056a with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:30.492676 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ddfa17abc4ad2a64d8015724b4ae4" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:30.492877 9796 raft_consensus.cc:3037] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.495726 9796 raft_consensus.cc:2454] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:30.496615 6866 leader_election.cc:302] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:30.496943 10729 raft_consensus.cc:2781] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.497078 10729 raft_consensus.cc:686] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:30.497371 10729 consensus_queue.cc:237] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:30.500427 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:30.503990 9663 raft_consensus.cc:1203] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:30.504063 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b3ddfa17abc4ad2a64d8015724b4ae4" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:30.504304 7061 raft_consensus.cc:3037] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.507359 7061 raft_consensus.cc:2454] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:30.510255 11388 leader_election.cc:288] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:30.516947 11463 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:30.518997 11493 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:30.523829 11464 tablet.cc:1722] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:30.525029 6655 catalog_manager.cc:5484] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:30.540120 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=6141d88e5e93405cabc65badf2126003] alter complete (version 3)
I20240503 14:13:30.641852 6650 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:30.642122 6650 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:30.646533 6650 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 413becdc566741348329e37bb740e9c1
I20240503 14:13:30.647274 10371 raft_consensus.cc:1050] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:30.647990 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 413becdc566741348329e37bb740e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:30.648200 10371 consensus_queue.cc:237] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:30.648533 11457 tablet_replica.cc:330] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:30.648767 11457 raft_consensus.cc:2229] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:30.649111 11457 raft_consensus.cc:2258] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.649657 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 413becdc566741348329e37bb740e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:30.650672 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 413becdc566741348329e37bb740e9c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:30 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:30.651257 11457 ts_tablet_manager.cc:1875] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.661952 11553 tablet_replica.cc:330] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:30.662735 11553 raft_consensus.cc:2229] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.663287 11553 raft_consensus.cc:2258] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.664484 11553 ts_tablet_manager.cc:1875] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.681367 11457 ts_tablet_manager.cc:1888] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:30.681581 11457 log.cc:1196] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/413becdc566741348329e37bb740e9c1
I20240503 14:13:30.682135 11457 ts_tablet_manager.cc:1909] T 413becdc566741348329e37bb740e9c1 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:30.683320 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table_new_name [id=6141d88e5e93405cabc65badf2126003]) successfully deleted
I20240503 14:13:30.687009 10729 raft_consensus.cc:1050] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:30.687870 10729 consensus_queue.cc:237] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:30.701167 9664 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.701509 11551 tablet_replica.cc:330] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:30.701833 11551 raft_consensus.cc:2229] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:30.702127 11551 raft_consensus.cc:2258] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:30.702226 11473 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.702637 11553 ts_tablet_manager.cc:1888] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:30.702764 11553 log.cc:1196] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/413becdc566741348329e37bb740e9c1
I20240503 14:13:30.703385 11553 ts_tablet_manager.cc:1909] T 413becdc566741348329e37bb740e9c1 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:30.704772 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table_new_name [id=6141d88e5e93405cabc65badf2126003]) successfully deleted
I20240503 14:13:30.705598 11551 ts_tablet_manager.cc:1875] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:30.710095 9795 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.710201 7061 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.711480 7189 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.712390 11551 ts_tablet_manager.cc:1888] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:30.712582 11551 log.cc:1196] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/413becdc566741348329e37bb740e9c1
I20240503 14:13:30.712646 11388 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.713148 11551 ts_tablet_manager.cc:1909] T 413becdc566741348329e37bb740e9c1 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:30.714428 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 413becdc566741348329e37bb740e9c1 (table rebalancer_dynamic_table_new_name [id=6141d88e5e93405cabc65badf2126003]) successfully deleted
I20240503 14:13:30.717469 7189 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.715598 9872 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240503 14:13:30.718858 9872 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.721258 11195 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.729748 10371 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.747112 11556 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.751921 6929 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:30.752882 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:30.769454 7060 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.775878 11195 consensus_queue.cc:1035] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.775689 10371 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:30.784261 7188 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:30.787953 9929 raft_consensus.cc:1203] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:30.788064 11473 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:30.788775 9796 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.793084 9663 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:30.794999 6655 catalog_manager.cc:5484] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:30.802206 6930 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:30.806378 7187 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:30.809866 6655 catalog_manager.cc:5484] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:30.814572 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 0811f5974a3b4acca9938c2e43318e9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]), partition=
I20240503 14:13:30.815007 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0811f5974a3b4acca9938c2e43318e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.815016 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 0811f5974a3b4acca9938c2e43318e9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]), partition=
I20240503 14:13:30.815354 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0811f5974a3b4acca9938c2e43318e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.821511 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 0811f5974a3b4acca9938c2e43318e9c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]), partition=
I20240503 14:13:30.821913 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0811f5974a3b4acca9938c2e43318e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:30.826159 11572 tablet_bootstrap.cc:492] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:30.828298 11572 tablet_bootstrap.cc:654] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.836445 11448 tablet_bootstrap.cc:492] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:30.837188 11572 tablet_bootstrap.cc:492] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:30.837486 11572 ts_tablet_manager.cc:1367] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 14:13:30.838342 11572 raft_consensus.cc:348] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.838667 11572 raft_consensus.cc:374] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.838771 11448 tablet_bootstrap.cc:654] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.838881 11572 raft_consensus.cc:729] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.839242 11572 consensus_queue.cc:260] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.841868 11572 ts_tablet_manager.cc:1398] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:30.848810 11118 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.875169 11573 tablet_bootstrap.cc:492] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:30.876730 7060 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac"
dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55256
I20240503 14:13:30.876987 7060 raft_consensus.cc:595] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:30.877869 11118 consensus_queue.cc:1035] T e478266c296c473eb7ebd1f5ebf7d5cf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.904706 11448 tablet_bootstrap.cc:492] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:30.904883 11448 ts_tablet_manager.cc:1367] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.069s user 0.007s sys 0.002s
I20240503 14:13:30.905670 11448 raft_consensus.cc:348] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.905892 11448 raft_consensus.cc:374] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.906076 11448 raft_consensus.cc:729] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.906373 11448 consensus_queue.cc:260] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.909480 11448 ts_tablet_manager.cc:1398] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:30.909471 11491 raft_consensus.cc:482] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:30.909718 11491 raft_consensus.cc:504] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.910627 11491 leader_election.cc:288] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:30.911079 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0811f5974a3b4acca9938c2e43318e9c" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:30.911227 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0811f5974a3b4acca9938c2e43318e9c" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:30.911644 9796 raft_consensus.cc:2454] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
W20240503 14:13:30.911871 9467 leader_election.cc:341] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:30.912078 9468 leader_election.cc:302] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters: f1a669e28f014fe287932acf56bebc13
I20240503 14:13:30.912693 11491 raft_consensus.cc:2781] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:30.912825 11491 raft_consensus.cc:482] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:30.912846 11573 tablet_bootstrap.cc:654] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:30.912932 11491 raft_consensus.cc:3037] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.918323 11491 raft_consensus.cc:504] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.921118 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0811f5974a3b4acca9938c2e43318e9c" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:30.921335 9796 raft_consensus.cc:3037] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:30.926098 9796 raft_consensus.cc:2454] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:30.926744 9468 leader_election.cc:302] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:30.927395 11585 raft_consensus.cc:2781] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:30.929199 11573 tablet_bootstrap.cc:492] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:30.929462 11573 ts_tablet_manager.cc:1367] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.007s
I20240503 14:13:30.929508 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0811f5974a3b4acca9938c2e43318e9c" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:30.930047 11491 leader_election.cc:288] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
W20240503 14:13:30.930600 9467 leader_election.cc:341] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:30.931501 11573 raft_consensus.cc:348] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.931833 11573 raft_consensus.cc:374] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:30.931993 11573 raft_consensus.cc:729] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:30.932359 11573 consensus_queue.cc:260] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.934414 11573 ts_tablet_manager.cc:1398] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:30.938971 10785 raft_consensus.cc:979] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: : Instructing follower e1bf5fb7402c46829974e2f74cd88708 to start an election
I20240503 14:13:30.939198 10785 raft_consensus.cc:1067] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Signalling peer e1bf5fb7402c46829974e2f74cd88708 to start an election
I20240503 14:13:30.941191 7191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac"
dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
from {username='slave'} at 127.0.186.3:41545
I20240503 14:13:30.941488 7191 raft_consensus.cc:482] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:30.941632 7191 raft_consensus.cc:3037] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:30.941620 11585 raft_consensus.cc:686] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:30.942093 11585 consensus_queue.cc:237] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:30.944875 7191 raft_consensus.cc:504] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:30.948756 7191 leader_election.cc:288] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:30.948705 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:30.948951 6930 raft_consensus.cc:3037] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:30.949066 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:30.949317 9663 raft_consensus.cc:3037] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:30.948947 11556 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 14:13:30.949524 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669d722d3eb1475dba616ed72cb775ac" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:30.949726 7060 raft_consensus.cc:3032] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:30.949820 7060 raft_consensus.cc:729] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:30.950157 7060 consensus_queue.cc:260] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:30.950681 7060 raft_consensus.cc:3037] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:30.952494 9663 raft_consensus.cc:2454] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 2.
I20240503 14:13:30.952769 6930 raft_consensus.cc:2454] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 2.
I20240503 14:13:30.953368 7125 leader_election.cc:302] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:30.953526 7060 raft_consensus.cc:2454] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 2.
I20240503 14:13:30.953737 10988 raft_consensus.cc:2781] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:30.953904 10988 raft_consensus.cc:686] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 2 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:30.954182 10988 consensus_queue.cc:237] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:30.960110 6655 catalog_manager.cc:5484] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 2 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:30.970292 11388 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:30.978405 6655 catalog_manager.cc:5484] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:31.004412 6930 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84"
dest_uuid: "7186ed8510424fd597173eaf10c57b12"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60482
I20240503 14:13:31.004667 6930 raft_consensus.cc:595] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:31.051131 7060 consensus_queue.cc:237] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:31.053560 7191 raft_consensus.cc:1261] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:31.053673 6800 raft_consensus.cc:1261] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:31.053784 6997 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09a755b35d694a2b89ca1f58f0fa7976. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:31.054299 11473 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:31.054581 10785 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:31.057505 10785 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:31.058883 6800 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:31.061769 7191 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:31.064690 6655 catalog_manager.cc:5484] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:31.081722 9530 consensus_queue.cc:237] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:31.084633 9929 raft_consensus.cc:1261] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.084810 9796 raft_consensus.cc:1261] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:31.084946 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62a55d0a1d7c4d7aba6635de1994e60b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:31.085489 11491 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:31.085819 11585 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:31.088778 11491 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:31.089929 9929 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:31.092056 9795 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:31.095273 10729 raft_consensus.cc:1050] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:31.096033 10729 consensus_queue.cc:237] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:31.097121 6654 catalog_manager.cc:5484] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:31.100478 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:31.108500 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:31.120702 6800 raft_consensus.cc:3037] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.122723 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 58548735201648fa805fdf6af390c088 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.123232 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58548735201648fa805fdf6af390c088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.124409 6800 raft_consensus.cc:1261] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.126628 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5ba4e0e3044c1f85816ed69abc9386 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.127108 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5ba4e0e3044c1f85816ed69abc9386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.127108 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 58548735201648fa805fdf6af390c088 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.127655 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58548735201648fa805fdf6af390c088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.132004 9643 tablet_service.cc:1467] Processing CreateTablet for tablet 1423a44f18494d7baece06af1d005ea0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.132416 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1423a44f18494d7baece06af1d005ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.132745 9796 raft_consensus.cc:1261] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.132771 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5ba4e0e3044c1f85816ed69abc9386 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.133183 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5ba4e0e3044c1f85816ed69abc9386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.135989 11586 consensus_queue.cc:1035] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.135972 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5ba4e0e3044c1f85816ed69abc9386 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.136665 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5ba4e0e3044c1f85816ed69abc9386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.136883 11586 consensus_queue.cc:1035] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.140482 9663 raft_consensus.cc:1261] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:31.145669 6655 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]). Expected version 1 got 0
I20240503 14:13:31.146099 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 1423a44f18494d7baece06af1d005ea0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.146679 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1423a44f18494d7baece06af1d005ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.149412 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 58548735201648fa805fdf6af390c088 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.149891 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58548735201648fa805fdf6af390c088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.150278 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 1423a44f18494d7baece06af1d005ea0 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:31.150430 11388 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:31.151288 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1423a44f18494d7baece06af1d005ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.151664 7060 raft_consensus.cc:1261] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:31.152058 9530 raft_consensus.cc:1261] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:31.153385 11388 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:31.157374 11388 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:31.161899 11388 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.165729 11448 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:31.167704 11448 tablet_bootstrap.cc:654] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.172163 7061 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.180676 11556 raft_consensus.cc:979] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: : Instructing follower 3efdbd3506274dba846ae40404f0572c to start an election
I20240503 14:13:31.180832 11556 raft_consensus.cc:1067] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Signalling peer 3efdbd3506274dba846ae40404f0572c to start an election
I20240503 14:13:31.182739 9530 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.186681 11642 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:31.187373 11642 tablet.cc:1620] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:31.190205 6658 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]). Expected version 1 got 0
I20240503 14:13:31.191746 9795 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84"
dest_uuid: "3efdbd3506274dba846ae40404f0572c"
from {username='slave'} at 127.0.186.2:50341
I20240503 14:13:31.192113 9795 raft_consensus.cc:482] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:31.192286 11647 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:31.192366 9795 raft_consensus.cc:3037] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:31.192054 6655 catalog_manager.cc:5484] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.195981 9795 raft_consensus.cc:504] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:31.198567 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:31.198772 6930 raft_consensus.cc:3032] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:31.198865 6930 raft_consensus.cc:729] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:31.199117 6930 consensus_queue.cc:260] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:31.199523 6930 raft_consensus.cc:3037] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:31.201081 11573 tablet_bootstrap.cc:492] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:31.201846 11572 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:31.202445 6930 raft_consensus.cc:2454] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 2.
I20240503 14:13:31.202788 11573 tablet_bootstrap.cc:654] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.203608 11572 tablet_bootstrap.cc:654] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.204197 9795 leader_election.cc:288] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:31.205063 11644 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:31.206204 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:31.206964 7061 raft_consensus.cc:3037] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:31.208379 6654 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]). Expected version 1 got 0
I20240503 14:13:31.197321 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c727d4554464442a6a81a8ad4996c84" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:31.211119 7191 raft_consensus.cc:3037] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:31.211328 7061 raft_consensus.cc:2454] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 2.
I20240503 14:13:31.214347 7191 raft_consensus.cc:2454] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 2.
I20240503 14:13:31.214586 11640 tablet_bootstrap.cc:492] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:31.216781 11640 tablet_bootstrap.cc:654] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.219451 9664 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.221767 11572 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:31.221974 11572 ts_tablet_manager.cc:1367] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240503 14:13:31.222856 11572 raft_consensus.cc:348] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.223217 11572 raft_consensus.cc:374] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.223348 11572 raft_consensus.cc:729] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.223690 11572 consensus_queue.cc:260] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.224371 11572 ts_tablet_manager.cc:1398] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:31.225661 11572 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:31.218636 9735 leader_election.cc:302] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 3efdbd3506274dba846ae40404f0572c, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:31.227531 11422 raft_consensus.cc:2781] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:31.227787 11572 tablet_bootstrap.cc:654] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.227782 11422 raft_consensus.cc:686] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 2 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:31.228132 11422 consensus_queue.cc:237] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:31.230227 6650 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]). Expected version 1 got 0
I20240503 14:13:31.233816 11644 tablet.cc:1620] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:31.238819 11647 tablet_bootstrap.cc:654] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.238632 7061 consensus_queue.cc:237] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:31.241463 9664 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:31.241595 6930 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:31.242261 10785 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.242848 10785 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.244278 6930 consensus_queue.cc:237] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:31.244938 11572 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:31.245091 11572 ts_tablet_manager.cc:1367] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240503 14:13:31.246399 9662 raft_consensus.cc:1261] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:31.246569 7061 raft_consensus.cc:1261] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:31.246979 10371 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.249203 11388 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:31.249348 11572 raft_consensus.cc:348] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.249547 11572 raft_consensus.cc:374] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.251170 11388 consensus_queue.cc:1035] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:31.249624 11572 raft_consensus.cc:729] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.252678 11572 consensus_queue.cc:260] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.253643 9665 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.254590 11572 ts_tablet_manager.cc:1398] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 14:13:31.256242 6930 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.258230 10729 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.264210 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a44ed92d65114bfda2e0f38054b17ada with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:31.264294 6655 catalog_manager.cc:5484] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 1 to 2, leader changed from 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 2 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:31.268932 6658 catalog_manager.cc:5484] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:31.274308 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9f30640fc81e4aeead38e3f8659c056a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:31.280071 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc] alter complete (version 1)
I20240503 14:13:31.280031 9665 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.284688 7058 consensus_queue.cc:237] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:31.287914 7060 raft_consensus.cc:2932] T 9f30640fc81e4aeead38e3f8659c056a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.218698 11448 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:31.291124 11448 ts_tablet_manager.cc:1367] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.125s user 0.005s sys 0.008s
I20240503 14:13:31.292085 11448 raft_consensus.cc:348] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.292392 11448 raft_consensus.cc:374] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.292476 11448 raft_consensus.cc:729] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.292783 11448 consensus_queue.cc:260] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.293418 11448 ts_tablet_manager.cc:1398] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:31.301374 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet a44ed92d65114bfda2e0f38054b17ada with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:31.304373 6650 catalog_manager.cc:5484] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:31.317090 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f30640fc81e4aeead38e3f8659c056a with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:31.319748 11225 raft_consensus.cc:482] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.319958 11225 raft_consensus.cc:504] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.320837 11225 leader_election.cc:288] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:31.321678 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58548735201648fa805fdf6af390c088" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:31.321897 9530 raft_consensus.cc:2454] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:31.322360 9735 leader_election.cc:302] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:31.322285 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58548735201648fa805fdf6af390c088" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
W20240503 14:13:31.323009 9733 leader_election.cc:341] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:31.323750 11225 raft_consensus.cc:2781] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.323851 11225 raft_consensus.cc:482] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.323948 11225 raft_consensus.cc:3037] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.330083 11225 raft_consensus.cc:504] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.331411 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58548735201648fa805fdf6af390c088" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:31.331681 9530 raft_consensus.cc:3037] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.332835 7191 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:31.333348 6930 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:31.333457 10371 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:31.333887 11473 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:31.334514 9530 raft_consensus.cc:2454] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
W20240503 14:13:31.335167 6995 consensus_peers.cc:485] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8848a40563c43468ac46f8c604cc578. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:31.335860 9735 leader_election.cc:302] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:31.336180 11422 raft_consensus.cc:2781] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.341991 11225 leader_election.cc:288] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:31.341926 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58548735201648fa805fdf6af390c088" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
W20240503 14:13:31.342912 9733 leader_election.cc:341] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:31.353040 10729 raft_consensus.cc:1050] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:31.353914 10729 consensus_queue.cc:237] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:31.369264 11573 tablet_bootstrap.cc:492] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:31.369382 9929 raft_consensus.cc:1261] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:31.369506 11573 ts_tablet_manager.cc:1367] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.168s user 0.010s sys 0.000s
I20240503 14:13:31.370131 11388 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:31.370432 11573 raft_consensus.cc:348] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.370715 11573 raft_consensus.cc:374] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.370800 11573 raft_consensus.cc:729] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.371066 11573 consensus_queue.cc:260] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.371862 11573 ts_tablet_manager.cc:1398] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:31.375250 11400 raft_consensus.cc:482] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.375419 11400 raft_consensus.cc:504] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.376259 11400 leader_election.cc:288] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:31.377341 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5ba4e0e3044c1f85816ed69abc9386" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:31.379073 7187 raft_consensus.cc:1261] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:31.379963 11388 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:31.384138 11637 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:31.387559 11637 tablet_bootstrap.cc:654] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.396770 7058 raft_consensus.cc:1261] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:31.397348 11388 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:31.398325 6734 leader_election.cc:341] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:31.401139 11659 tablet_replica.cc:330] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:31.401484 11659 raft_consensus.cc:2229] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.403383 11659 raft_consensus.cc:2258] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.406337 11422 raft_consensus.cc:686] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:31.407198 11422 consensus_queue.cc:237] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.408188 11388 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:31.411958 6650 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]). Expected version 1 got 0
I20240503 14:13:31.414745 7187 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:31.415212 7058 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:31.415714 11659 ts_tablet_manager.cc:1875] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:31.415982 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:31.405633 11661 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:31.417343 11661 tablet.cc:1620] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:31.424342 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5ba4e0e3044c1f85816ed69abc9386" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:31.423241 9929 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
W20240503 14:13:31.426162 6734 leader_election.cc:341] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:31.422833 6651 catalog_manager.cc:5484] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:31.426314 6734 leader_election.cc:302] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1a669e28f014fe287932acf56bebc13; no voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:31.427750 11400 raft_consensus.cc:2726] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:31.428082 6655 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]). Expected version 2 got 1
I20240503 14:13:31.430414 11644 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:31.430585 7061 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:31.430886 11644 tablet.cc:1620] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:31.435380 6929 consensus_queue.cc:237] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:31.436739 11132 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.440407 7187 raft_consensus.cc:1261] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:31.441567 11634 raft_consensus.cc:482] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.441758 11634 raft_consensus.cc:504] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.442685 11634 leader_election.cc:288] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:31.442860 11556 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.443475 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423a44f18494d7baece06af1d005ea0" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:31.444919 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423a44f18494d7baece06af1d005ea0" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
W20240503 14:13:31.445501 9467 leader_election.cc:341] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:13:31.445777 9468 leader_election.cc:341] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:31.445911 9468 leader_election.cc:302] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:31.446899 11635 raft_consensus.cc:2726] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:31.447083 11659 ts_tablet_manager.cc:1888] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:31.446565 10729 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:31.451809 11473 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:31.446975 7188 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:31.455552 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f56e8ea68eb94475838da245b5f060cc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:31.456498 11642 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:31.463344 11659 log.cc:1196] T a44ed92d65114bfda2e0f38054b17ada P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/a44ed92d65114bfda2e0f38054b17ada
I20240503 14:13:31.463800 7188 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:31.464696 6651 catalog_manager.cc:5484] T f56e8ea68eb94475838da245b5f060cc P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:31.465860 11642 tablet.cc:1620] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:31.470638 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet f56e8ea68eb94475838da245b5f060cc with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:31.470834 9664 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:31.473578 10988 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.477145 6929 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:31.478235 6651 catalog_manager.cc:5484] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:31.488623 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet a44ed92d65114bfda2e0f38054b17ada (table rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]) successfully deleted
I20240503 14:13:31.499926 11676 tablet_replica.cc:330] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:31.504989 11637 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:31.505287 11637 ts_tablet_manager.cc:1367] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.121s user 0.002s sys 0.002s
I20240503 14:13:31.505086 11676 raft_consensus.cc:2229] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.506292 11637 raft_consensus.cc:348] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.506531 11637 raft_consensus.cc:374] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.506620 11637 raft_consensus.cc:729] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.508231 11676 raft_consensus.cc:2258] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.513067 11676 ts_tablet_manager.cc:1875] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:31.514240 9928 raft_consensus.cc:1261] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:31.515120 11388 consensus_queue.cc:1035] T f56e8ea68eb94475838da245b5f060cc P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.517045 9928 raft_consensus.cc:2932] T f56e8ea68eb94475838da245b5f060cc P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:31.521301 6650 catalog_manager.cc:5484] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:31.527230 6930 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:31.528390 11660 tablet_replica.cc:330] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:31.529884 10988 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.533164 11660 raft_consensus.cc:2229] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.534376 11676 ts_tablet_manager.cc:1888] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:31.534529 11676 log.cc:1196] T f56e8ea68eb94475838da245b5f060cc P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/f56e8ea68eb94475838da245b5f060cc
I20240503 14:13:31.536619 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet f56e8ea68eb94475838da245b5f060cc (table rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]) successfully deleted
I20240503 14:13:31.537386 11637 consensus_queue.cc:260] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:31.539052 11637 ts_tablet_manager.cc:1398] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.034s user 0.000s sys 0.003s
I20240503 14:13:31.541584 11637 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:31.543870 11640 tablet_bootstrap.cc:492] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:31.546150 11637 tablet_bootstrap.cc:654] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.546233 11640 ts_tablet_manager.cc:1367] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.332s user 0.044s sys 0.054s
I20240503 14:13:31.548024 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc] alter complete (version 2)
I20240503 14:13:31.549831 11640 raft_consensus.cc:348] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.553196 11640 raft_consensus.cc:374] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.553927 11640 raft_consensus.cc:729] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.554776 11640 consensus_queue.cc:260] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.555205 11637 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:31.555553 11637 ts_tablet_manager.cc:1367] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.005s
I20240503 14:13:31.556464 11640 ts_tablet_manager.cc:1398] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:31.556958 11647 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:31.557147 11647 ts_tablet_manager.cc:1367] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.365s user 0.017s sys 0.034s
I20240503 14:13:31.557981 11647 raft_consensus.cc:348] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.558202 11647 raft_consensus.cc:374] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.559057 11647 raft_consensus.cc:729] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.559399 11647 consensus_queue.cc:260] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.563202 11647 ts_tablet_manager.cc:1398] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240503 14:13:31.564450 11637 raft_consensus.cc:348] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.567035 11637 raft_consensus.cc:374] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.567821 11660 raft_consensus.cc:2258] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.572235 11660 ts_tablet_manager.cc:1875] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:31.571123 11637 raft_consensus.cc:729] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.581528 6650 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table [id=ce5e85c93b624d62ba1160d1e5e163dc]). Expected version 2 got 1
I20240503 14:13:31.583745 11661 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:31.596308 11661 tablet.cc:1620] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:31.575544 11637 consensus_queue.cc:260] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.610088 11691 ts_tablet_manager.cc:902] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:31.614138 11637 ts_tablet_manager.cc:1398] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.058s user 0.003s sys 0.000s
I20240503 14:13:31.614346 11689 ts_tablet_manager.cc:902] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:31.617168 11637 tablet_bootstrap.cc:492] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:31.623531 7188 consensus_queue.cc:237] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:31.625074 11118 raft_consensus.cc:482] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.625279 11637 tablet_bootstrap.cc:654] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.625264 11118 raft_consensus.cc:504] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.625927 11118 leader_election.cc:288] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:31.627152 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5ba4e0e3044c1f85816ed69abc9386" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:31.622792 11660 ts_tablet_manager.cc:1888] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:31.627439 11660 log.cc:1196] T 9f30640fc81e4aeead38e3f8659c056a P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/9f30640fc81e4aeead38e3f8659c056a
I20240503 14:13:31.627406 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5ba4e0e3044c1f85816ed69abc9386" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:31.627661 6800 raft_consensus.cc:2454] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:31.628307 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 9f30640fc81e4aeead38e3f8659c056a (table rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]) successfully deleted
I20240503 14:13:31.634394 9863 leader_election.cc:302] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters:
W20240503 14:13:31.634696 9863 leader_election.cc:341] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:31.635582 11118 raft_consensus.cc:2781] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.635699 11118 raft_consensus.cc:482] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.635855 11118 raft_consensus.cc:3037] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.638559 11118 raft_consensus.cc:504] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.639159 11118 leader_election.cc:288] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:31.640169 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5ba4e0e3044c1f85816ed69abc9386" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:31.640743 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5ba4e0e3044c1f85816ed69abc9386" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:31.640965 6800 raft_consensus.cc:3037] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:31.641350 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 652f2bf72ec9462384afcf9854509767. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:31.641557 9797 raft_consensus.cc:1261] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.642488 10988 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:31.644832 11132 raft_consensus.cc:482] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.644990 9664 raft_consensus.cc:1261] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.644999 11132 raft_consensus.cc:504] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
W20240503 14:13:31.645370 9863 leader_election.cc:341] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:31.645753 11132 leader_election.cc:288] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:31.645906 11132 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:31.646205 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423a44f18494d7baece06af1d005ea0" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:31.646389 11637 tablet_bootstrap.cc:492] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:31.646436 9664 raft_consensus.cc:2454] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:31.646560 11637 ts_tablet_manager.cc:1367] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.002s
I20240503 14:13:31.646967 7125 leader_election.cc:302] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:31.647367 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423a44f18494d7baece06af1d005ea0" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:31.647367 11637 raft_consensus.cc:348] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.647639 9530 raft_consensus.cc:2454] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:31.647861 6800 raft_consensus.cc:2454] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:31.648464 9863 leader_election.cc:302] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:31.648978 11132 raft_consensus.cc:2781] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:31.649101 11132 raft_consensus.cc:482] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:31.649231 11132 raft_consensus.cc:3037] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.651902 11118 raft_consensus.cc:2781] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.652487 11118 raft_consensus.cc:686] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:31.652829 11118 consensus_queue.cc:237] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.654505 11637 raft_consensus.cc:374] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:31.655001 11637 raft_consensus.cc:729] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:31.656180 11637 consensus_queue.cc:260] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.656363 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:31.656450 11132 raft_consensus.cc:504] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.657613 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423a44f18494d7baece06af1d005ea0" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:31.657822 9530 raft_consensus.cc:3037] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.659052 11637 ts_tablet_manager.cc:1398] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 14:13:31.660705 9530 raft_consensus.cc:2454] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:31.661247 7127 leader_election.cc:302] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:31.661592 10988 raft_consensus.cc:2781] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:31.666162 11132 leader_election.cc:288] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:31.666374 11642 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.666571 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1423a44f18494d7baece06af1d005ea0" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:31.666795 9665 raft_consensus.cc:3037] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.669816 9665 raft_consensus.cc:2454] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:31.673609 11644 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.677419 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=ce5e85c93b624d62ba1160d1e5e163dc] alter complete (version 3)
I20240503 14:13:31.687232 11690 tablet.cc:1722] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:31.703171 9795 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:31.704175 10988 raft_consensus.cc:686] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:31.704507 10988 consensus_queue.cc:237] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:31.706481 11691 tablet_copy_client.cc:308] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.5:36609
I20240503 14:13:31.707620 9553 tablet_copy_service.cc:140] P 5b7e36bf354f480c90859cb8867b59cf: Received BeginTabletCopySession request for tablet 62a55d0a1d7c4d7aba6635de1994e60b from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:55131)
I20240503 14:13:31.707931 9553 tablet_copy_service.cc:161] P 5b7e36bf354f480c90859cb8867b59cf: Beginning new tablet copy session on tablet 62a55d0a1d7c4d7aba6635de1994e60b from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:55131: session id = 09723f236d8044538d920a37b5d9aa8c-62a55d0a1d7c4d7aba6635de1994e60b
I20240503 14:13:31.703519 11702 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:31.714753 9553 tablet_copy_source_session.cc:205] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:31.717047 6655 catalog_manager.cc:5484] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:31.719374 11689 tablet_copy_client.cc:308] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:31.719098 6651 catalog_manager.cc:5484] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:31.720939 11691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a55d0a1d7c4d7aba6635de1994e60b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.724349 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 09a755b35d694a2b89ca1f58f0fa7976 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:53967)
I20240503 14:13:31.724495 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 09a755b35d694a2b89ca1f58f0fa7976 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:53967: session id = 5b7e36bf354f480c90859cb8867b59cf-09a755b35d694a2b89ca1f58f0fa7976
I20240503 14:13:31.728015 6651 catalog_manager.cc:5484] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:31.729667 7081 tablet_copy_source_session.cc:205] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:31.729324 9665 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:31.736317 11691 tablet_copy_client.cc:783] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:31.736692 11691 tablet_copy_client.cc:647] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:31.738984 11691 tablet_copy_client.cc:523] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:31.741925 6929 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:31.742759 11422 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.743332 11689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a755b35d694a2b89ca1f58f0fa7976. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.749732 11691 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:31.760030 11689 tablet_copy_client.cc:783] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:31.761102 11689 tablet_copy_client.cc:647] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:31.762773 11689 tablet_copy_client.cc:523] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:31.766469 11689 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:31.779765 6651 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:31.780081 6651 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:31.784250 6651 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 0811f5974a3b4acca9938c2e43318e9c
I20240503 14:13:31.787498 11691 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:31.788206 11691 tablet_bootstrap.cc:492] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:31.789090 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 0811f5974a3b4acca9938c2e43318e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:31.789402 11660 tablet_replica.cc:330] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:31.789618 11660 raft_consensus.cc:2229] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:31.790066 11660 raft_consensus.cc:2258] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.790392 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 0811f5974a3b4acca9938c2e43318e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:31.790527 11691 ts_tablet_manager.cc:1367] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.003s
I20240503 14:13:31.790604 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 0811f5974a3b4acca9938c2e43318e9c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:31 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:31.791568 11691 raft_consensus.cc:348] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:31.791851 11691 raft_consensus.cc:729] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:31.792181 11691 consensus_queue.cc:260] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:31.798388 11660 ts_tablet_manager.cc:1875] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.799293 11691 ts_tablet_manager.cc:1398] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 14:13:31.800114 9553 tablet_copy_service.cc:342] P 5b7e36bf354f480c90859cb8867b59cf: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-62a55d0a1d7c4d7aba6635de1994e60b received from {username='slave'} at 127.0.186.3:55131
I20240503 14:13:31.800323 9553 tablet_copy_service.cc:433] P 5b7e36bf354f480c90859cb8867b59cf: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-62a55d0a1d7c4d7aba6635de1994e60b on tablet 62a55d0a1d7c4d7aba6635de1994e60b with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:31.803567 7061 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:31.805286 11664 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.812350 7188 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:31.813164 11225 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:31.818677 11716 tablet_replica.cc:330] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:31.823011 11716 raft_consensus.cc:2229] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.823439 11716 raft_consensus.cc:2258] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.824571 11716 ts_tablet_manager.cc:1875] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.831372 11660 ts_tablet_manager.cc:1888] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:31.831641 11660 log.cc:1196] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/0811f5974a3b4acca9938c2e43318e9c
I20240503 14:13:31.832247 11660 ts_tablet_manager.cc:1909] T 0811f5974a3b4acca9938c2e43318e9c P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:31.840543 11717 ts_tablet_manager.cc:902] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:31.845930 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table_new_name [id=ce5e85c93b624d62ba1160d1e5e163dc]) successfully deleted
I20240503 14:13:31.849490 11715 tablet_replica.cc:330] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:31.849745 11715 raft_consensus.cc:2229] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:31.849987 11715 raft_consensus.cc:2258] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:31.851382 11716 ts_tablet_manager.cc:1888] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:31.851537 11716 log.cc:1196] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/0811f5974a3b4acca9938c2e43318e9c
I20240503 14:13:31.852540 11715 ts_tablet_manager.cc:1875] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:31.853617 11716 ts_tablet_manager.cc:1909] T 0811f5974a3b4acca9938c2e43318e9c P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:31.859932 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table_new_name [id=ce5e85c93b624d62ba1160d1e5e163dc]) successfully deleted
I20240503 14:13:31.863572 11564 raft_consensus.cc:482] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:31.863780 11564 raft_consensus.cc:504] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:31.864511 11564 leader_election.cc:288] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:31.865427 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5ba4e0e3044c1f85816ed69abc9386" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:31.865726 6800 raft_consensus.cc:2379] T 3e5ba4e0e3044c1f85816ed69abc9386 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f03dc37fe3234033bc665bed4f1b51f5 in current term 1: Already voted for candidate ea801601713a4d55aee0de9e3473e0bc in this term.
I20240503 14:13:31.866423 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5ba4e0e3044c1f85816ed69abc9386" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:31.867949 9599 leader_election.cc:302] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5; no voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:31.869920 11564 raft_consensus.cc:3037] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.873044 11564 raft_consensus.cc:2726] T 3e5ba4e0e3044c1f85816ed69abc9386 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:31.876649 11715 ts_tablet_manager.cc:1888] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:31.877195 11715 log.cc:1196] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/0811f5974a3b4acca9938c2e43318e9c
I20240503 14:13:31.878352 11715 ts_tablet_manager.cc:1909] T 0811f5974a3b4acca9938c2e43318e9c P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:31.880478 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 0811f5974a3b4acca9938c2e43318e9c (table rebalancer_dynamic_table_new_name [id=ce5e85c93b624d62ba1160d1e5e163dc]) successfully deleted
I20240503 14:13:31.882586 11717 tablet_copy_client.cc:308] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:31.883322 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet c8848a40563c43468ac46f8c604cc578 from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:42011)
I20240503 14:13:31.883508 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet c8848a40563c43468ac46f8c604cc578 from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:42011: session id = ea801601713a4d55aee0de9e3473e0bc-c8848a40563c43468ac46f8c604cc578
I20240503 14:13:31.883831 9928 consensus_queue.cc:237] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:31.886842 7081 tablet_copy_source_session.cc:205] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:31.891089 11717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8848a40563c43468ac46f8c604cc578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.896798 11689 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:31.897287 11689 tablet_bootstrap.cc:492] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:31.898604 11717 tablet_copy_client.cc:783] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:31.898680 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:31.898916 11717 tablet_copy_client.cc:647] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:31.902801 11689 ts_tablet_manager.cc:1367] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.136s user 0.050s sys 0.009s
I20240503 14:13:31.903681 9664 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.903795 11689 raft_consensus.cc:348] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:31.904228 11689 raft_consensus.cc:729] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:31.904543 11118 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:31.904536 11689 consensus_queue.cc:260] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
W20240503 14:13:31.905169 9864 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32d91b6f4bb244c78bc6fef8b837f964. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:31.902330 9530 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:31.905738 11689 ts_tablet_manager.cc:1398] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:31.906481 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-09a755b35d694a2b89ca1f58f0fa7976 received from {username='slave'} at 127.0.186.5:53967
I20240503 14:13:31.906627 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-09a755b35d694a2b89ca1f58f0fa7976 on tablet 09a755b35d694a2b89ca1f58f0fa7976 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:31.909312 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 7e834ab7d29145dc96ee190255b15c68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f5c5e0f519cb4716974153f5fe6a970d]), partition=
I20240503 14:13:31.909724 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e834ab7d29145dc96ee190255b15c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.909965 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 7e834ab7d29145dc96ee190255b15c68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f5c5e0f519cb4716974153f5fe6a970d]), partition=
I20240503 14:13:31.910091 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 7e834ab7d29145dc96ee190255b15c68 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f5c5e0f519cb4716974153f5fe6a970d]), partition=
I20240503 14:13:31.910310 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e834ab7d29145dc96ee190255b15c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.910427 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e834ab7d29145dc96ee190255b15c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:31.911060 11118 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:31.917338 9665 raft_consensus.cc:3037] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:31.918565 11664 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.929983 11664 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:31.932664 11726 tablet_bootstrap.cc:492] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:31.935565 11726 tablet_bootstrap.cc:654] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.936079 11647 tablet_bootstrap.cc:492] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:31.937723 11647 tablet_bootstrap.cc:654] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:31.966831 7188 consensus_queue.cc:237] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:31.971918 9662 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:31.973881 11702 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:31.983170 6929 raft_consensus.cc:1261] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:31.984424 11702 consensus_queue.cc:1035] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:31.985622 11717 tablet_copy_client.cc:523] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:31.986758 11702 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.994097 9662 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:31.997716 11717 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:31.999361 6929 raft_consensus.cc:2932] T 669d722d3eb1475dba616ed72cb775ac P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:32.000411 9530 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:31.975574 11118 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:32.008031 11725 tablet_bootstrap.cc:492] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:32.009850 11725 tablet_bootstrap.cc:654] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.012650 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 669d722d3eb1475dba616ed72cb775ac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:32.013330 9662 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:32.013934 6650 catalog_manager.cc:5484] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 5 to 7, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 2 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:32.019894 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 669d722d3eb1475dba616ed72cb775ac with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:32.020269 11676 tablet_replica.cc:330] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:32.020506 11676 raft_consensus.cc:2229] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.020717 11676 raft_consensus.cc:2258] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.021047 11647 tablet_bootstrap.cc:492] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:32.021212 11647 ts_tablet_manager.cc:1367] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.006s
I20240503 14:13:32.021909 11676 ts_tablet_manager.cc:1875] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:32.022084 11647 raft_consensus.cc:348] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.022293 11647 raft_consensus.cc:374] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.022372 11647 raft_consensus.cc:729] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.022614 11647 consensus_queue.cc:260] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.023209 11647 ts_tablet_manager.cc:1398] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:32.027750 9530 raft_consensus.cc:1203] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:32.035300 11726 tablet_bootstrap.cc:492] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:32.035127 6654 catalog_manager.cc:5484] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:32.035480 11726 ts_tablet_manager.cc:1367] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.103s user 0.042s sys 0.015s
I20240503 14:13:32.036437 11726 raft_consensus.cc:348] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.037000 11726 raft_consensus.cc:374] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.037163 11726 raft_consensus.cc:729] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.037942 11726 consensus_queue.cc:260] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.038635 11726 ts_tablet_manager.cc:1398] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:32.043488 11676 ts_tablet_manager.cc:1888] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:13:32.043808 11676 log.cc:1196] T 669d722d3eb1475dba616ed72cb775ac P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/669d722d3eb1475dba616ed72cb775ac
I20240503 14:13:32.045284 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 669d722d3eb1475dba616ed72cb775ac (table rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]) successfully deleted
I20240503 14:13:32.050105 11731 consensus_queue.cc:1035] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.053308 11725 tablet_bootstrap.cc:492] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:32.053543 11725 ts_tablet_manager.cc:1367] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240503 14:13:32.054695 11725 raft_consensus.cc:348] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.054997 11725 raft_consensus.cc:374] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.055106 11725 raft_consensus.cc:729] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.055390 11725 consensus_queue.cc:260] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.056134 11725 ts_tablet_manager.cc:1398] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 14:13:32.076318 11118 consensus_queue.cc:1035] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.111616 7188 consensus_queue.cc:237] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:32.114109 11717 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:32.114620 11717 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:32.114929 11717 ts_tablet_manager.cc:1367] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.117s user 0.047s sys 0.007s
I20240503 14:13:32.115496 9664 raft_consensus.cc:1261] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:32.115769 11717 raft_consensus.cc:348] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:32.115978 11717 raft_consensus.cc:729] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
W20240503 14:13:32.116179 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e5410b22c57427d96a32fc9957cea8e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:32.116196 11717 consensus_queue.cc:260] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:32.116375 10988 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:32.116923 11717 ts_tablet_manager.cc:1398] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:32.117725 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-c8848a40563c43468ac46f8c604cc578 received from {username='slave'} at 127.0.186.8:42011
I20240503 14:13:32.117856 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-c8848a40563c43468ac46f8c604cc578 on tablet c8848a40563c43468ac46f8c604cc578 with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:32.121810 9530 raft_consensus.cc:1261] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:32.121809 10988 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:32.126209 11707 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:32.128064 9664 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:32.129966 9530 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:32.131855 6655 catalog_manager.cc:5484] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:32.139452 11586 raft_consensus.cc:482] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.139662 11586 raft_consensus.cc:504] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.140447 11586 leader_election.cc:288] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:32.141187 7188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e834ab7d29145dc96ee190255b15c68" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:32.141419 7188 raft_consensus.cc:2454] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:32.141819 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e834ab7d29145dc96ee190255b15c68" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:32.141880 9468 leader_election.cc:302] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:32.142040 9797 raft_consensus.cc:2454] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:32.142166 11586 raft_consensus.cc:2781] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:32.142264 11586 raft_consensus.cc:482] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:32.142361 11586 raft_consensus.cc:3037] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.145658 11586 raft_consensus.cc:504] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.146600 11586 leader_election.cc:288] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:32.147485 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e834ab7d29145dc96ee190255b15c68" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:32.147962 9797 raft_consensus.cc:3037] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.148069 7188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e834ab7d29145dc96ee190255b15c68" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:32.148366 7188 raft_consensus.cc:3037] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.152603 9797 raft_consensus.cc:2454] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:32.153077 9468 leader_election.cc:302] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:32.155138 11586 raft_consensus.cc:2781] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:32.157363 11586 raft_consensus.cc:686] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:32.157951 11586 consensus_queue.cc:237] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:32.159909 7188 raft_consensus.cc:2454] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:32.164908 6655 catalog_manager.cc:5484] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:32.179167 7061 raft_consensus.cc:1203] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:32.185868 7188 consensus_queue.cc:237] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:32.189492 7058 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:32.189672 6929 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:32.189911 11691 ts_tablet_manager.cc:902] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:32.190119 11132 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:32.191161 11132 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:32.192895 7126 consensus_peers.cc:485] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b4363edcc954769a3ab9c658e198c2d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:32.202333 9797 consensus_queue.cc:237] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:32.205227 7061 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:32.206221 11225 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:32.209241 7188 raft_consensus.cc:1261] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:32.209632 11225 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:32.210808 11132 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:32.211700 11664 consensus_queue.cc:1035] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:32.213054 7061 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:32.217319 6929 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:32.219782 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c727d4554464442a6a81a8ad4996c84 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:32.222448 6655 catalog_manager.cc:5484] T 1c727d4554464442a6a81a8ad4996c84 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:32.223140 11691 tablet_copy_client.cc:308] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:32.224664 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 652f2bf72ec9462384afcf9854509767 from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:41545)
I20240503 14:13:32.224854 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 652f2bf72ec9462384afcf9854509767 from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:41545: session id = 09723f236d8044538d920a37b5d9aa8c-652f2bf72ec9462384afcf9854509767
I20240503 14:13:32.227452 7211 tablet_copy_source_session.cc:205] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:32.229113 11691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652f2bf72ec9462384afcf9854509767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.229172 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c727d4554464442a6a81a8ad4996c84 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:32.246127 11759 tablet_replica.cc:330] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:32.246568 11759 raft_consensus.cc:2229] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.246901 11759 raft_consensus.cc:2258] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.246891 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:32.248181 11759 ts_tablet_manager.cc:1875] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:32.249440 6654 catalog_manager.cc:5484] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:32.257278 11691 tablet_copy_client.cc:783] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:32.257570 11691 tablet_copy_client.cc:647] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:32.257545 11759 ts_tablet_manager.cc:1888] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:13:32.257669 11759 log.cc:1196] T 1c727d4554464442a6a81a8ad4996c84 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/1c727d4554464442a6a81a8ad4996c84
I20240503 14:13:32.258464 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 1c727d4554464442a6a81a8ad4996c84 (table rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]) successfully deleted
I20240503 14:13:32.262085 11691 tablet_copy_client.cc:523] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:32.268059 11691 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:32.228575 7188 raft_consensus.cc:2932] T 1c727d4554464442a6a81a8ad4996c84 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:32.279630 11132 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.299819 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:32.313179 11691 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:32.314322 11691 tablet_bootstrap.cc:492] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:32.314898 11691 ts_tablet_manager.cc:1367] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.000s
I20240503 14:13:32.315821 11691 raft_consensus.cc:348] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:32.316079 11691 raft_consensus.cc:729] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:32.319775 11691 consensus_queue.cc:260] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:32.320761 11691 ts_tablet_manager.cc:1398] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:32.324470 10988 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.332386 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-652f2bf72ec9462384afcf9854509767 received from {username='slave'} at 127.0.186.3:41545
I20240503 14:13:32.332587 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-652f2bf72ec9462384afcf9854509767 on tablet 652f2bf72ec9462384afcf9854509767 with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:32.352937 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 9a89b1b988df4ad8aa8f1311cfa35731 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.353055 9797 raft_consensus.cc:1261] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:32.353440 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a89b1b988df4ad8aa8f1311cfa35731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.353579 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 9a89b1b988df4ad8aa8f1311cfa35731 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.353839 11586 consensus_queue.cc:1035] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.354022 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a89b1b988df4ad8aa8f1311cfa35731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.354681 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 9a89b1b988df4ad8aa8f1311cfa35731 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.355098 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a89b1b988df4ad8aa8f1311cfa35731. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.357496 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 603002fdf6b84d3ea63c65636eb42cb1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.357913 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603002fdf6b84d3ea63c65636eb42cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.358268 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 13f41f236987487b8bca287c7ec1fdd3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.358682 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f41f236987487b8bca287c7ec1fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.364091 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 13f41f236987487b8bca287c7ec1fdd3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.364642 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f41f236987487b8bca287c7ec1fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.364842 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 13f41f236987487b8bca287c7ec1fdd3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.365258 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f41f236987487b8bca287c7ec1fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.367363 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 603002fdf6b84d3ea63c65636eb42cb1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.367756 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603002fdf6b84d3ea63c65636eb42cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.371517 11725 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:32.372032 9643 tablet_service.cc:1467] Processing CreateTablet for tablet 603002fdf6b84d3ea63c65636eb42cb1 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:32.372437 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603002fdf6b84d3ea63c65636eb42cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.373430 11725 tablet_bootstrap.cc:654] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.376750 11726 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:32.378571 11726 tablet_bootstrap.cc:654] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.386324 7191 raft_consensus.cc:1261] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:32.387001 11586 consensus_queue.cc:1035] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.387243 7061 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:32.391388 11726 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:32.391671 11726 ts_tablet_manager.cc:1367] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.001s
I20240503 14:13:32.392597 11726 raft_consensus.cc:348] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:32.392833 9929 raft_consensus.cc:1203] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:32.392927 11726 raft_consensus.cc:374] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.393002 11726 raft_consensus.cc:729] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.393258 11726 consensus_queue.cc:260] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:32.394033 11726 ts_tablet_manager.cc:1398] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:32.396443 11725 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:32.396615 11725 ts_tablet_manager.cc:1367] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20240503 14:13:32.397441 11725 raft_consensus.cc:348] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.397675 11725 raft_consensus.cc:374] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.397751 11725 raft_consensus.cc:729] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.397994 11725 consensus_queue.cc:260] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.398626 11725 ts_tablet_manager.cc:1398] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:32.399196 11771 tablet_bootstrap.cc:492] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:32.400795 11771 tablet_bootstrap.cc:654] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.408030 6654 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 7e834ab7d29145dc96ee190255b15c68 (table rebalancer_dynamic_table [id=f5c5e0f519cb4716974153f5fe6a970d]). Expected version 1 got 0
I20240503 14:13:32.449070 11771 tablet_bootstrap.cc:492] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:32.449378 11771 ts_tablet_manager.cc:1367] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.050s user 0.033s sys 0.011s
I20240503 14:13:32.450493 11771 raft_consensus.cc:348] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.450889 11771 raft_consensus.cc:374] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.451064 11771 raft_consensus.cc:729] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.451378 11771 consensus_queue.cc:260] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.453583 11771 ts_tablet_manager.cc:1398] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:13:32.453905 11771 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:32.454661 6800 consensus_queue.cc:237] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:32.455883 11771 tablet_bootstrap.cc:654] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.465812 11780 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:32.466575 11780 tablet.cc:1620] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:32.468430 11782 tablet_bootstrap.cc:492] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:32.470398 11782 tablet_bootstrap.cc:654] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.476405 11772 tablet_bootstrap.cc:492] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:32.476682 11779 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:32.478168 11779 tablet.cc:1620] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:32.481109 11772 tablet_bootstrap.cc:654] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.488358 6658 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 7e834ab7d29145dc96ee190255b15c68 (table rebalancer_dynamic_table [id=f5c5e0f519cb4716974153f5fe6a970d]). Expected version 1 got 0
I20240503 14:13:32.498368 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=f5c5e0f519cb4716974153f5fe6a970d] alter complete (version 1)
I20240503 14:13:32.498652 11787 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:32.499264 11787 tablet.cc:1620] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
W20240503 14:13:32.504681 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcb23bc5b0a44da7989aa828826ece07. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:32.505424 11772 tablet_bootstrap.cc:492] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:32.505578 7058 raft_consensus.cc:1261] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:32.505635 6929 raft_consensus.cc:1261] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:32.505699 11772 ts_tablet_manager.cc:1367] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.005s
I20240503 14:13:32.506278 11400 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:32.506763 11772 raft_consensus.cc:348] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.506958 11772 raft_consensus.cc:374] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.507063 11772 raft_consensus.cc:729] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.507365 11772 consensus_queue.cc:260] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.508352 11400 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:32.503197 6650 catalog_manager.cc:5576] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337) does not have the latest schema for tablet 7e834ab7d29145dc96ee190255b15c68 (table rebalancer_dynamic_table [id=f5c5e0f519cb4716974153f5fe6a970d]). Expected version 1 got 0
I20240503 14:13:32.531507 11564 raft_consensus.cc:482] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.531746 11564 raft_consensus.cc:504] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.532531 11564 leader_election.cc:288] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:32.533358 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a89b1b988df4ad8aa8f1311cfa35731" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:32.533581 9929 raft_consensus.cc:2454] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:32.533353 11400 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:32.534411 9599 leader_election.cc:302] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:32.535475 11564 raft_consensus.cc:2781] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:32.535569 11564 raft_consensus.cc:482] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:32.535681 11564 raft_consensus.cc:3037] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.536468 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a89b1b988df4ad8aa8f1311cfa35731" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
W20240503 14:13:32.537349 9599 leader_election.cc:341] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:32.538466 11564 raft_consensus.cc:504] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.538105 6930 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:32.540011 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a89b1b988df4ad8aa8f1311cfa35731" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:32.540208 9929 raft_consensus.cc:3037] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.542547 9929 raft_consensus.cc:2454] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:32.543277 9599 leader_election.cc:302] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:32.544528 11564 leader_election.cc:288] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:32.544711 11564 raft_consensus.cc:2781] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:32.544842 11564 raft_consensus.cc:686] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:32.545076 11564 consensus_queue.cc:237] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.551579 6654 catalog_manager.cc:5484] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:32.552704 6653 catalog_manager.cc:5484] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:32.560550 11586 raft_consensus.cc:482] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.560828 11586 raft_consensus.cc:504] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:32.562462 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:32.562886 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
W20240503 14:13:32.563199 9467 leader_election.cc:341] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:13:32.563463 9467 leader_election.cc:341] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:32.563573 9467 leader_election.cc:302] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:32.563868 11634 raft_consensus.cc:2726] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:32.564271 11586 leader_election.cc:288] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:32.575884 11634 raft_consensus.cc:1050] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:32.576786 11772 ts_tablet_manager.cc:1398] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.071s user 0.001s sys 0.002s
I20240503 14:13:32.577126 11772 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:32.577085 11634 consensus_queue.cc:237] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:32.578886 11772 tablet_bootstrap.cc:654] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.579985 11782 tablet_bootstrap.cc:492] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:32.580147 11782 ts_tablet_manager.cc:1367] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.000s
I20240503 14:13:32.580775 11782 raft_consensus.cc:348] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.580982 11782 raft_consensus.cc:374] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.581095 11782 raft_consensus.cc:729] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.581367 11782 consensus_queue.cc:260] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:32.584383 11782 ts_tablet_manager.cc:1398] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:32.585747 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a89b1b988df4ad8aa8f1311cfa35731" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:32.585983 6800 raft_consensus.cc:3037] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.589303 6800 raft_consensus.cc:2454] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:32.589967 11793 ts_tablet_manager.cc:902] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:32.592748 7058 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:32.595191 9929 raft_consensus.cc:1261] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:32.597745 7061 raft_consensus.cc:1261] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:32.599717 9797 raft_consensus.cc:1261] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:32.600610 11586 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:32.601591 11786 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:32.607306 11772 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:32.607501 11772 ts_tablet_manager.cc:1367] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.001s
I20240503 14:13:32.608160 11772 raft_consensus.cc:348] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.608862 11772 raft_consensus.cc:374] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.608987 11772 raft_consensus.cc:729] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.609320 11772 consensus_queue.cc:260] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.609933 11772 ts_tablet_manager.cc:1398] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:32.610612 11786 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:32.610944 11772 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:32.612854 11772 tablet_bootstrap.cc:654] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.613796 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:32.628504 11780 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:32.629109 11780 tablet.cc:1620] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:32.631755 11787 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:32.632290 11787 tablet.cc:1620] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:32.632896 11779 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:32.633456 11779 tablet.cc:1620] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:32.636003 11422 raft_consensus.cc:482] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:32.636059 11775 raft_consensus.cc:1050] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:32.636158 11422 raft_consensus.cc:504] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.636900 11422 leader_election.cc:288] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:32.636938 11775 consensus_queue.cc:237] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:32.637614 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f41f236987487b8bca287c7ec1fdd3" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
W20240503 14:13:32.638278 9733 leader_election.cc:341] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:32.639940 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f41f236987487b8bca287c7ec1fdd3" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:32.640195 9664 raft_consensus.cc:2454] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:32.640666 9733 leader_election.cc:302] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters: ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:32.641135 11422 raft_consensus.cc:2781] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:32.641237 11422 raft_consensus.cc:482] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:32.641319 11422 raft_consensus.cc:3037] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.641701 11771 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:32.641860 11771 ts_tablet_manager.cc:1367] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.188s user 0.017s sys 0.036s
I20240503 14:13:32.642643 11771 raft_consensus.cc:348] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:32.642865 11771 raft_consensus.cc:374] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.642927 11771 raft_consensus.cc:729] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.643142 11771 consensus_queue.cc:260] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:32.649220 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=f5c5e0f519cb4716974153f5fe6a970d] alter complete (version 2)
I20240503 14:13:32.651288 11772 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:32.651465 11772 ts_tablet_manager.cc:1367] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.003s
I20240503 14:13:32.651629 11771 ts_tablet_manager.cc:1398] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240503 14:13:32.653765 11771 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:32.653406 11422 raft_consensus.cc:504] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.655470 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f41f236987487b8bca287c7ec1fdd3" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:32.654736 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f41f236987487b8bca287c7ec1fdd3" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
W20240503 14:13:32.656424 9733 leader_election.cc:341] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:32.656677 11422 leader_election.cc:288] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:32.658933 11801 ts_tablet_manager.cc:902] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:32.658694 11586 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.660056 9664 raft_consensus.cc:3037] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:32.662458 7061 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.662878 9664 raft_consensus.cc:2454] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:32.663369 9733 leader_election.cc:302] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters: ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:32.663925 11772 raft_consensus.cc:348] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:32.664490 11422 raft_consensus.cc:2781] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:32.664547 11772 raft_consensus.cc:374] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.664647 11772 raft_consensus.cc:729] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.664894 7060 raft_consensus.cc:1203] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:32.664853 11772 consensus_queue.cc:260] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:32.665467 11422 raft_consensus.cc:686] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:32.665766 11422 consensus_queue.cc:237] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.666838 11772 ts_tablet_manager.cc:1398] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240503 14:13:32.668330 9929 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.674543 11771 tablet_bootstrap.cc:654] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:32.675025 6654 catalog_manager.cc:5484] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.685829 9530 consensus_queue.cc:237] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:32.693636 11771 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:32.693886 11771 ts_tablet_manager.cc:1367] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.007s
I20240503 14:13:32.694804 11771 raft_consensus.cc:348] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.697027 11771 raft_consensus.cc:374] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:32.697257 11771 raft_consensus.cc:729] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:32.697681 11771 consensus_queue.cc:260] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:32.701176 9797 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.701682 9929 raft_consensus.cc:1261] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:32.702823 7058 raft_consensus.cc:1261] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:32.703840 11799 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:32.706632 11786 consensus_queue.cc:1035] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:32.707607 11806 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.708580 7058 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.713881 11771 ts_tablet_manager.cc:1398] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240503 14:13:32.716193 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62a55d0a1d7c4d7aba6635de1994e60b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:32.717553 9929 raft_consensus.cc:2932] T 62a55d0a1d7c4d7aba6635de1994e60b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.720839 6650 catalog_manager.cc:5484] T 62a55d0a1d7c4d7aba6635de1994e60b P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:32.721302 6800 raft_consensus.cc:1261] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:32.721879 9530 raft_consensus.cc:1261] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:32.721951 11777 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:32.727041 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 62a55d0a1d7c4d7aba6635de1994e60b with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:32.728020 11777 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:32.735482 7188 raft_consensus.cc:1261] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:32.738940 11777 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:32.745468 11777 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:32.752029 7188 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:32.754004 9531 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:32.758445 6658 catalog_manager.cc:5484] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:32.761011 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:32.771648 11779 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:32.778457 11787 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:32.793457 11780 tablet.cc:1722] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:32.800148 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=f5c5e0f519cb4716974153f5fe6a970d] alter complete (version 3)
W20240503 14:13:32.848495 9735 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.5:36609, user_credentials={real_user=slave}} blocked reactor thread for 160412us
I20240503 14:13:32.848810 11820 tablet_replica.cc:330] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:32.850520 6658 catalog_manager.cc:5484] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:32.851740 6799 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:32.856456 7058 consensus_queue.cc:237] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:32.859163 9530 raft_consensus.cc:1261] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:32.859872 11473 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:32.862327 6799 raft_consensus.cc:1261] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:32.862201 10371 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:32.863044 11473 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:32.863164 9531 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:32.865913 6798 raft_consensus.cc:2932] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:32.866753 11818 ts_tablet_manager.cc:902] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:32.870695 11820 raft_consensus.cc:2229] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.871040 11820 raft_consensus.cc:2258] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.872366 6654 catalog_manager.cc:5484] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 5 to 6, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:32.872643 11820 ts_tablet_manager.cc:1875] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:32.876173 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a755b35d694a2b89ca1f58f0fa7976 with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:32.876526 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09a755b35d694a2b89ca1f58f0fa7976 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:32.886946 11818 tablet_copy_client.cc:308] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:32.887910 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 4b4363edcc954769a3ab9c658e198c2d from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:36385)
I20240503 14:13:32.888110 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 4b4363edcc954769a3ab9c658e198c2d from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:36385: session id = 3efdbd3506274dba846ae40404f0572c-4b4363edcc954769a3ab9c658e198c2d
I20240503 14:13:32.891731 11826 tablet_replica.cc:330] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:32.891986 11826 raft_consensus.cc:2229] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.892256 11826 raft_consensus.cc:2258] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.892849 7211 tablet_copy_source_session.cc:205] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:32.895385 11818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4363edcc954769a3ab9c658e198c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.895650 11793 tablet_copy_client.cc:308] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:32.896658 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet 32d91b6f4bb244c78bc6fef8b837f964 from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:48587)
I20240503 14:13:32.896843 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet 32d91b6f4bb244c78bc6fef8b837f964 from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:48587: session id = e1bf5fb7402c46829974e2f74cd88708-32d91b6f4bb244c78bc6fef8b837f964
I20240503 14:13:32.899641 9950 tablet_copy_source_session.cc:205] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:32.902501 11793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d91b6f4bb244c78bc6fef8b837f964. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:32.907980 6654 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:32.908224 6654 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:32.910329 11826 ts_tablet_manager.cc:1875] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:32.910209 11820 ts_tablet_manager.cc:1888] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:32.910619 11820 log.cc:1196] T 62a55d0a1d7c4d7aba6635de1994e60b P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/62a55d0a1d7c4d7aba6635de1994e60b
I20240503 14:13:32.911962 6654 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 7e834ab7d29145dc96ee190255b15c68
I20240503 14:13:32.912097 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 62a55d0a1d7c4d7aba6635de1994e60b (table rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]) successfully deleted
I20240503 14:13:32.912297 11801 tablet_copy_client.cc:308] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:32.913460 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e834ab7d29145dc96ee190255b15c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:32.913709 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e834ab7d29145dc96ee190255b15c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:32.918469 11818 tablet_copy_client.cc:783] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:32.918838 11818 tablet_copy_client.cc:647] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:32.919205 11833 tablet_replica.cc:330] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:32.919432 11833 raft_consensus.cc:2229] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:32.919847 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e834ab7d29145dc96ee190255b15c68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:32 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:32.919894 11833 raft_consensus.cc:2258] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.920198 11820 tablet_replica.cc:330] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:32.920375 11820 raft_consensus.cc:2229] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:32.920574 11820 raft_consensus.cc:2258] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:32.921511 11820 ts_tablet_manager.cc:1875] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:32.924160 11833 ts_tablet_manager.cc:1875] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:32.931141 11833 ts_tablet_manager.cc:1888] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:32.931354 11833 log.cc:1196] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/7e834ab7d29145dc96ee190255b15c68
I20240503 14:13:32.931252 11820 ts_tablet_manager.cc:1888] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:32.931537 11820 log.cc:1196] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/7e834ab7d29145dc96ee190255b15c68
I20240503 14:13:32.931864 11833 ts_tablet_manager.cc:1909] T 7e834ab7d29145dc96ee190255b15c68 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:32.932595 11820 ts_tablet_manager.cc:1909] T 7e834ab7d29145dc96ee190255b15c68 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:32.933086 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 7e834ab7d29145dc96ee190255b15c68 (table rebalancer_dynamic_table_new_name [id=f5c5e0f519cb4716974153f5fe6a970d]) successfully deleted
I20240503 14:13:32.934888 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 7e834ab7d29145dc96ee190255b15c68 (table rebalancer_dynamic_table_new_name [id=f5c5e0f519cb4716974153f5fe6a970d]) successfully deleted
I20240503 14:13:32.936367 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 7e5410b22c57427d96a32fc9957cea8e from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:43871)
I20240503 14:13:32.938918 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 7e5410b22c57427d96a32fc9957cea8e from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:43871: session id = 7186ed8510424fd597173eaf10c57b12-7e5410b22c57427d96a32fc9957cea8e
I20240503 14:13:32.950747 11777 raft_consensus.cc:1050] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:32.951704 11777 consensus_queue.cc:237] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:32.958891 9929 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:32.959815 11775 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:32.961484 6930 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:32.962059 11775 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:32.964212 11791 consensus_queue.cc:1035] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:32.967475 11818 tablet_copy_client.cc:523] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:32.969475 7211 tablet_copy_source_session.cc:205] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:32.971087 11818 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:32.982142 11793 tablet_copy_client.cc:783] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:32.982791 11793 tablet_copy_client.cc:647] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:32.986320 11793 tablet_copy_client.cc:523] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:32.994624 11793 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:32.995174 7188 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:32.995883 11801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e5410b22c57427d96a32fc9957cea8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.001996 11564 consensus_queue.cc:1035] T 9a89b1b988df4ad8aa8f1311cfa35731 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.004209 11801 tablet_copy_client.cc:783] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:33.004571 11801 tablet_copy_client.cc:647] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:33.012768 11777 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:33.020392 11473 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.022936 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:33.024570 6930 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.032377 11801 tablet_copy_client.cc:523] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:33.038431 11801 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:33.038745 11826 ts_tablet_manager.cc:1888] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:33.038928 11826 log.cc:1196] T 09a755b35d694a2b89ca1f58f0fa7976 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/09a755b35d694a2b89ca1f58f0fa7976
I20240503 14:13:33.048736 11826 tablet_replica.cc:330] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:33.048391 6658 catalog_manager.cc:5484] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.049011 11826 raft_consensus.cc:2229] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:33.051096 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 09a755b35d694a2b89ca1f58f0fa7976 (table rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]) successfully deleted
I20240503 14:13:33.054569 9928 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.058190 11818 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:33.059315 11818 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:33.059777 11818 ts_tablet_manager.cc:1367] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.089s user 0.061s sys 0.008s
I20240503 14:13:33.060077 11564 raft_consensus.cc:482] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.060272 11564 raft_consensus.cc:504] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:33.061064 11564 leader_election.cc:288] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:33.061040 11818 raft_consensus.cc:348] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:33.061372 11818 raft_consensus.cc:729] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:33.061533 11849 ts_tablet_manager.cc:902] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:33.061633 11818 consensus_queue.cc:260] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:33.062592 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:33.062873 9929 raft_consensus.cc:2454] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:33.063926 9599 leader_election.cc:302] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:33.067286 11818 ts_tablet_manager.cc:1398] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 14:13:33.069811 11826 raft_consensus.cc:2258] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.070369 11564 raft_consensus.cc:2781] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.070498 11564 raft_consensus.cc:482] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.070641 11564 raft_consensus.cc:3037] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.071944 11826 ts_tablet_manager.cc:1875] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:33.071851 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:33.072120 9531 raft_consensus.cc:2454] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:33.074904 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-4b4363edcc954769a3ab9c658e198c2d received from {username='slave'} at 127.0.186.7:36385
I20240503 14:13:33.075125 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-4b4363edcc954769a3ab9c658e198c2d on tablet 4b4363edcc954769a3ab9c658e198c2d with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:33.077658 9511 tablet_service.cc:1467] Processing CreateTablet for tablet f0611c8f61fe4ac2bb94818c7a804c97 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6]), partition=
I20240503 14:13:33.077824 9779 tablet_service.cc:1467] Processing CreateTablet for tablet f0611c8f61fe4ac2bb94818c7a804c97 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6]), partition=
I20240503 14:13:33.078083 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0611c8f61fe4ac2bb94818c7a804c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.078189 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0611c8f61fe4ac2bb94818c7a804c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.074481 11564 raft_consensus.cc:504] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:33.081665 9644 tablet_service.cc:1467] Processing CreateTablet for tablet f0611c8f61fe4ac2bb94818c7a804c97 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6]), partition=
I20240503 14:13:33.082093 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0611c8f61fe4ac2bb94818c7a804c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.083112 11564 leader_election.cc:288] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:33.083405 10988 raft_consensus.cc:1050] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:33.084348 10988 consensus_queue.cc:237] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:33.086768 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:33.087018 9531 raft_consensus.cc:3037] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.091261 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:33.091838 9928 raft_consensus.cc:3037] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.094997 9928 raft_consensus.cc:2454] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:33.096210 9531 raft_consensus.cc:2454] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:33.096786 9601 leader_election.cc:302] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:33.097115 11564 raft_consensus.cc:2781] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.097304 11564 raft_consensus.cc:686] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:33.097569 11564 consensus_queue.cc:237] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:33.100446 7058 consensus_queue.cc:237] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:33.097841 7188 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.113647 11772 tablet_bootstrap.cc:492] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:33.102866 9929 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:33.116490 11777 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:33.117581 11772 tablet_bootstrap.cc:654] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.121109 11801 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:33.126191 11801 tablet_bootstrap.cc:492] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:33.126065 11777 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.128165 11801 ts_tablet_manager.cc:1367] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.007s
I20240503 14:13:33.129048 11801 raft_consensus.cc:348] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:33.129292 11801 raft_consensus.cc:729] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:33.129513 11801 consensus_queue.cc:260] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:33.130152 11801 ts_tablet_manager.cc:1398] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:33.131947 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8848a40563c43468ac46f8c604cc578 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:33.135151 9929 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.139691 11826 ts_tablet_manager.cc:1888] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.139847 11826 log.cc:1196] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/7e834ab7d29145dc96ee190255b15c68
I20240503 14:13:33.141541 7191 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:33.143903 11775 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:33.145298 11857 tablet_bootstrap.cc:492] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:33.140880 11826 ts_tablet_manager.cc:1909] T 7e834ab7d29145dc96ee190255b15c68 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:33.147040 11857 tablet_bootstrap.cc:654] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.150193 7191 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.152277 11772 tablet_bootstrap.cc:492] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:33.152457 11772 ts_tablet_manager.cc:1367] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.009s
I20240503 14:13:33.153215 11772 raft_consensus.cc:348] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.153445 11772 raft_consensus.cc:374] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.153517 11772 raft_consensus.cc:729] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.153585 6651 catalog_manager.cc:5484] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:33.153753 11772 consensus_queue.cc:260] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.154289 11772 ts_tablet_manager.cc:1398] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:33.154507 6658 catalog_manager.cc:5484] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.156051 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-7e5410b22c57427d96a32fc9957cea8e received from {username='slave'} at 127.0.186.2:43871
I20240503 14:13:33.156172 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-7e5410b22c57427d96a32fc9957cea8e on tablet 7e5410b22c57427d96a32fc9957cea8e with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:33.157413 11854 tablet_bootstrap.cc:492] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:33.162695 11854 tablet_bootstrap.cc:654] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.166466 9662 raft_consensus.cc:1261] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:33.168903 11118 raft_consensus.cc:482] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.169096 11118 raft_consensus.cc:504] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.169802 11118 leader_election.cc:288] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:33.169811 11132 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:33.170675 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet c8848a40563c43468ac46f8c604cc578 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:33.170987 9795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f41f236987487b8bca287c7ec1fdd3" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:33.172458 9929 raft_consensus.cc:3037] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.174405 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13f41f236987487b8bca287c7ec1fdd3" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:33.174674 9664 raft_consensus.cc:2379] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea801601713a4d55aee0de9e3473e0bc in current term 1: Already voted for candidate 3efdbd3506274dba846ae40404f0572c in this term.
I20240503 14:13:33.178891 7061 raft_consensus.cc:1261] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:33.178917 9795 raft_consensus.cc:1261] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:33.179566 11707 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:33.180038 11664 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.180860 11132 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:33.186353 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 7e834ab7d29145dc96ee190255b15c68 (table rebalancer_dynamic_table_new_name [id=f5c5e0f519cb4716974153f5fe6a970d]) successfully deleted
I20240503 14:13:33.189591 11865 tablet_replica.cc:330] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:33.191089 11865 raft_consensus.cc:2229] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:33.197098 11849 tablet_copy_client.cc:308] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:34101
I20240503 14:13:33.197947 6820 tablet_copy_service.cc:140] P f1a669e28f014fe287932acf56bebc13: Received BeginTabletCopySession request for tablet dcb23bc5b0a44da7989aa828826ece07 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:35165)
I20240503 14:13:33.198130 6820 tablet_copy_service.cc:161] P f1a669e28f014fe287932acf56bebc13: Beginning new tablet copy session on tablet dcb23bc5b0a44da7989aa828826ece07 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:35165: session id = 5b7e36bf354f480c90859cb8867b59cf-dcb23bc5b0a44da7989aa828826ece07
I20240503 14:13:33.198526 9863 leader_election.cc:302] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:33.201082 11731 raft_consensus.cc:2726] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:33.201478 6820 tablet_copy_source_session.cc:205] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:33.203212 11849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcb23bc5b0a44da7989aa828826ece07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.210196 11793 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:33.211205 11865 raft_consensus.cc:2258] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.211957 11793 tablet_bootstrap.cc:492] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:33.212798 11865 ts_tablet_manager.cc:1875] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:33.213255 11793 ts_tablet_manager.cc:1367] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.219s user 0.013s sys 0.005s
I20240503 14:13:33.214407 11793 raft_consensus.cc:348] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:33.216040 11793 raft_consensus.cc:729] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:33.217331 11857 tablet_bootstrap.cc:492] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:33.217367 11793 consensus_queue.cc:260] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:33.218091 11793 ts_tablet_manager.cc:1398] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:33.218137 11664 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.220417 11854 tablet_bootstrap.cc:492] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:33.220073 11707 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.220639 11854 ts_tablet_manager.cc:1367] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.002s
I20240503 14:13:33.221325 11857 ts_tablet_manager.cc:1367] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.007s
I20240503 14:13:33.222090 11854 raft_consensus.cc:348] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.222287 11854 raft_consensus.cc:374] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.222373 11854 raft_consensus.cc:729] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.222641 11854 consensus_queue.cc:260] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.224256 11854 ts_tablet_manager.cc:1398] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:33.224733 11857 raft_consensus.cc:348] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.224965 11857 raft_consensus.cc:374] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.225069 11857 raft_consensus.cc:729] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.228883 11857 consensus_queue.cc:260] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.233022 11422 raft_consensus.cc:482] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:33.233227 11422 raft_consensus.cc:504] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.233922 11422 leader_election.cc:288] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:33.234670 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0611c8f61fe4ac2bb94818c7a804c97" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:33.234949 9531 raft_consensus.cc:2454] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:33.235499 9735 leader_election.cc:302] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:33.235646 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0611c8f61fe4ac2bb94818c7a804c97" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:33.235926 9662 raft_consensus.cc:2454] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:33.236204 11422 raft_consensus.cc:2781] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:33.236328 11422 raft_consensus.cc:482] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:33.236459 11422 raft_consensus.cc:3037] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.236533 11857 ts_tablet_manager.cc:1398] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 14:13:33.238960 11849 tablet_copy_client.cc:783] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:33.239248 11849 tablet_copy_client.cc:647] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:33.239202 11422 raft_consensus.cc:504] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.240422 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0611c8f61fe4ac2bb94818c7a804c97" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:33.240607 9531 raft_consensus.cc:3037] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.240535 9795 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.243363 9531 raft_consensus.cc:2454] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:33.244035 9735 leader_election.cc:302] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:33.244320 11664 raft_consensus.cc:2781] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:33.243852 9662 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.244474 11664 raft_consensus.cc:686] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:33.244715 11422 leader_election.cc:288] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:33.244802 11664 consensus_queue.cc:237] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:33.246361 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0611c8f61fe4ac2bb94818c7a804c97" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:33.246590 9664 raft_consensus.cc:3037] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:33.250388 9664 raft_consensus.cc:2454] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:33.251560 7058 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.256923 11849 tablet_copy_client.cc:523] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:33.261008 6654 catalog_manager.cc:5484] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.271039 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-32d91b6f4bb244c78bc6fef8b837f964 received from {username='slave'} at 127.0.186.4:48587
I20240503 14:13:33.271189 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-32d91b6f4bb244c78bc6fef8b837f964 on tablet 32d91b6f4bb244c78bc6fef8b837f964 with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:33.276546 11849 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:33.288290 6658 catalog_manager.cc:5484] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:33.304026 7126 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 58948.8us
I20240503 14:13:33.305518 9797 raft_consensus.cc:1203] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:33.313561 7191 consensus_queue.cc:237] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:33.326843 7060 raft_consensus.cc:1261] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:33.328785 9795 raft_consensus.cc:1261] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:33.329164 11865 ts_tablet_manager.cc:1888] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:33.329300 11865 log.cc:1196] T c8848a40563c43468ac46f8c604cc578 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/c8848a40563c43468ac46f8c604cc578
I20240503 14:13:33.330318 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet c8848a40563c43468ac46f8c604cc578 (table rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]) successfully deleted
I20240503 14:13:33.334537 11874 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:33.335654 11874 consensus_queue.cc:1035] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:33.343798 11872 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.359532 7060 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.359812 9795 raft_consensus.cc:2932] T 652f2bf72ec9462384afcf9854509767 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.365691 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 652f2bf72ec9462384afcf9854509767 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:33.366880 6654 catalog_manager.cc:5484] T 652f2bf72ec9462384afcf9854509767 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:33.371340 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 652f2bf72ec9462384afcf9854509767 with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:33.384588 11883 tablet_replica.cc:330] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:33.384963 11883 raft_consensus.cc:2229] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:33.385362 11883 raft_consensus.cc:2258] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.386489 11883 ts_tablet_manager.cc:1875] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:33.403745 11883 ts_tablet_manager.cc:1888] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:33.404023 11883 log.cc:1196] T 652f2bf72ec9462384afcf9854509767 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/652f2bf72ec9462384afcf9854509767
I20240503 14:13:33.406905 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 652f2bf72ec9462384afcf9854509767 (table rebalancer_extra_table_6 [id=3535c9e6d3d6450ea394b4bb9367476b]) successfully deleted
I20240503 14:13:33.415894 11849 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:33.416404 11849 tablet_bootstrap.cc:492] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:33.416707 11849 ts_tablet_manager.cc:1367] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.140s user 0.077s sys 0.037s
I20240503 14:13:33.417469 11849 raft_consensus.cc:348] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:33.417748 11849 raft_consensus.cc:729] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:33.417943 11849 consensus_queue.cc:260] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:33.418256 6653 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:33.418581 11849 ts_tablet_manager.cc:1398] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:33.419821 6820 tablet_copy_service.cc:342] P f1a669e28f014fe287932acf56bebc13: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-dcb23bc5b0a44da7989aa828826ece07 received from {username='slave'} at 127.0.186.5:35165
I20240503 14:13:33.419963 6820 tablet_copy_service.cc:433] P f1a669e28f014fe287932acf56bebc13: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-dcb23bc5b0a44da7989aa828826ece07 on tablet dcb23bc5b0a44da7989aa828826ece07 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:33.425894 9530 raft_consensus.cc:1261] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.425894 9662 raft_consensus.cc:1261] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.426467 11664 consensus_queue.cc:1035] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.426965 11664 consensus_queue.cc:1035] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.434259 6653 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet f0611c8f61fe4ac2bb94818c7a804c97 (table rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6]). Expected version 1 got 0
I20240503 14:13:33.435631 11890 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:33.436723 11890 tablet.cc:1620] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:33.437482 6653 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet f0611c8f61fe4ac2bb94818c7a804c97 (table rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6]). Expected version 1 got 0
I20240503 14:13:33.443317 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6] alter complete (version 1)
I20240503 14:13:33.443651 11891 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:33.444171 11891 tablet.cc:1620] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:33.444671 6653 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet f0611c8f61fe4ac2bb94818c7a804c97 (table rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6]). Expected version 1 got 0
I20240503 14:13:33.449482 11892 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:33.451166 6653 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet f0611c8f61fe4ac2bb94818c7a804c97 (table rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6]). Expected version 1 got 0
I20240503 14:13:33.451759 11892 tablet.cc:1620] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:33.454298 6930 raft_consensus.cc:1203] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:33.498497 11564 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.507344 11564 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:33.548550 7191 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32"
dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50214
I20240503 14:13:33.548854 7191 raft_consensus.cc:595] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:33.551887 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:33.560227 11890 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:33.560812 11890 tablet.cc:1620] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:33.561211 11892 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:33.562314 11892 tablet.cc:1620] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:33.563800 11891 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:33.564299 11891 tablet.cc:1620] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:33.569988 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=fe456347dbd74c3c809e0d38df0987d6] alter complete (version 2)
I20240503 14:13:33.646972 6930 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:33.649657 7191 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:33.649958 6865 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b154f780fc742fa9b0efd609c307f95. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:33.650575 11815 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:33.657917 11815 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:33.664490 7191 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:33.670038 6654 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:33.671813 6653 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:33.685806 11878 raft_consensus.cc:979] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: : Instructing follower 09723f236d8044538d920a37b5d9aa8c to start an election
I20240503 14:13:33.685971 11878 raft_consensus.cc:1067] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Signalling peer 09723f236d8044538d920a37b5d9aa8c to start an election
I20240503 14:13:33.686965 7060 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32"
dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
from {username='slave'} at 127.0.186.4:38199
I20240503 14:13:33.687225 7060 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:33.687350 7060 raft_consensus.cc:3037] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:33.690812 7060 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:33.692890 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:33.693116 6929 raft_consensus.cc:3037] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:33.696367 11890 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:33.699103 11891 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:33.708529 11892 tablet.cc:1722] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:33.715883 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=fe456347dbd74c3c809e0d38df0987d6] alter complete (version 3)
I20240503 14:13:33.715972 6929 raft_consensus.cc:2454] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 2.
I20240503 14:13:33.716696 7060 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:33.717190 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:33.717504 9531 raft_consensus.cc:3037] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:33.718711 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d3ec4f0833d4d489f774757312efb32" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:33.718933 7191 raft_consensus.cc:3032] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:33.719049 7191 raft_consensus.cc:729] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:33.719758 7191 consensus_queue.cc:260] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:33.720346 7191 raft_consensus.cc:3037] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:33.721724 9531 raft_consensus.cc:2454] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 2.
I20240503 14:13:33.722505 6997 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:33.722914 11775 raft_consensus.cc:2781] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:33.723079 11775 raft_consensus.cc:686] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:33.723352 11775 consensus_queue.cc:237] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:33.725450 7060 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:33.731869 11969 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:33.735487 7191 raft_consensus.cc:2454] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 2.
I20240503 14:13:33.764875 7188 raft_consensus.cc:1203] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:33.780334 7060 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:33.785375 11874 raft_consensus.cc:1050] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:33.786523 11874 consensus_queue.cc:237] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
W20240503 14:13:33.787864 6865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 125402us
I20240503 14:13:33.789193 6651 catalog_manager.cc:5484] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 1 to 2, leader changed from e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 2 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.805922 9795 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:33.806737 11872 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:33.813100 6930 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:33.816572 11876 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:33.819581 6651 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:33.819835 6651 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:33.823868 6651 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet f0611c8f61fe4ac2bb94818c7a804c97
I20240503 14:13:33.825284 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet f0611c8f61fe4ac2bb94818c7a804c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:33 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:33.826697 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet f0611c8f61fe4ac2bb94818c7a804c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:33 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:33.827071 11883 tablet_replica.cc:330] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:33.827323 11883 raft_consensus.cc:2229] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:33.827576 11883 raft_consensus.cc:2258] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.828698 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet f0611c8f61fe4ac2bb94818c7a804c97 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:33 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:33.829985 7058 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:33.830353 11883 ts_tablet_manager.cc:1875] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:33.831084 11874 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240503 14:13:33.837829 11883 ts_tablet_manager.cc:1888] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.838078 11883 log.cc:1196] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/f0611c8f61fe4ac2bb94818c7a804c97
I20240503 14:13:33.838681 11883 ts_tablet_manager.cc:1909] T f0611c8f61fe4ac2bb94818c7a804c97 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:33.840112 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet f0611c8f61fe4ac2bb94818c7a804c97 (table rebalancer_dynamic_table_new_name [id=fe456347dbd74c3c809e0d38df0987d6]) successfully deleted
I20240503 14:13:33.844549 11984 tablet_replica.cc:330] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:33.845048 11985 tablet_replica.cc:330] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:33.846154 11985 raft_consensus.cc:2229] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:33.846491 11984 raft_consensus.cc:2229] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:33.846977 11985 raft_consensus.cc:2258] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.847318 11984 raft_consensus.cc:2258] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:33.849916 11985 ts_tablet_manager.cc:1875] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:33.850711 11984 ts_tablet_manager.cc:1875] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:33.857718 11876 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:33.858249 9531 raft_consensus.cc:1203] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:33.859949 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:33.874125 6930 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:33.875039 9644 tablet_service.cc:1467] Processing CreateTablet for tablet df05d420e5cd426e88cd33de16a707f6 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.875571 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df05d420e5cd426e88cd33de16a707f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.876121 6653 catalog_manager.cc:5484] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:33.878650 6910 tablet_service.cc:1467] Processing CreateTablet for tablet df05d420e5cd426e88cd33de16a707f6 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.879096 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df05d420e5cd426e88cd33de16a707f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.879405 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 89ff0991a4e245609405ff945388417b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.882009 7058 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:33.884106 11988 tablet_bootstrap.cc:492] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:33.884011 9910 tablet_service.cc:1467] Processing CreateTablet for tablet df05d420e5cd426e88cd33de16a707f6 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.884423 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df05d420e5cd426e88cd33de16a707f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.884407 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 89ff0991a4e245609405ff945388417b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.884696 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ff0991a4e245609405ff945388417b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.886658 9797 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:33.891871 11984 ts_tablet_manager.cc:1888] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:33.892194 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 89ff0991a4e245609405ff945388417b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.892685 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ff0991a4e245609405ff945388417b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.896549 7188 consensus_queue.cc:237] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:33.898582 11984 log.cc:1196] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/f0611c8f61fe4ac2bb94818c7a804c97
I20240503 14:13:33.899298 11984 ts_tablet_manager.cc:1909] T f0611c8f61fe4ac2bb94818c7a804c97 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:33.900511 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet f0611c8f61fe4ac2bb94818c7a804c97 (table rebalancer_dynamic_table_new_name [id=fe456347dbd74c3c809e0d38df0987d6]) successfully deleted
I20240503 14:13:33.900826 6911 tablet_service.cc:1467] Processing CreateTablet for tablet f4b86d126b92400f9b14a9e4fb527fc9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.901337 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b86d126b92400f9b14a9e4fb527fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.906210 9797 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:33.907459 11872 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:33.910918 6930 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:33.911571 11872 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:33.913126 9908 tablet_service.cc:1467] Processing CreateTablet for tablet f4b86d126b92400f9b14a9e4fb527fc9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.913579 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b86d126b92400f9b14a9e4fb527fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.914258 9511 tablet_service.cc:1467] Processing CreateTablet for tablet f4b86d126b92400f9b14a9e4fb527fc9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:33.914762 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b86d126b92400f9b14a9e4fb527fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.920167 11990 tablet_bootstrap.cc:492] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:33.921289 11878 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:33.924244 11990 tablet_bootstrap.cc:654] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.926127 11989 tablet_bootstrap.cc:492] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:33.925985 6929 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:33.928231 11989 tablet_bootstrap.cc:654] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.928546 9530 consensus_queue.cc:237] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:33.931371 9797 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:33.931478 11989 tablet_bootstrap.cc:492] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:33.931632 11989 ts_tablet_manager.cc:1367] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:33.932089 11806 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:33.932428 11989 raft_consensus.cc:348] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:33.932623 11989 raft_consensus.cc:374] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:33.932732 11989 raft_consensus.cc:729] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:33.933226 11989 consensus_queue.cc:260] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:33.935191 11872 raft_consensus.cc:1050] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:33.935945 11989 ts_tablet_manager.cc:1398] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:33.936314 11989 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:33.936295 11872 consensus_queue.cc:237] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
W20240503 14:13:33.939777 9467 consensus_peers.cc:485] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9641f7f5a3e4debb5aabc6b8a7497f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:33.940822 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:33.948083 11989 tablet_bootstrap.cc:654] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:33.949893 11806 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:33.956110 6658 catalog_manager.cc:5484] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:33.959008 9797 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:33.968119 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb79d46e3614382a451c6c021a8d997 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f2259e021ef4a648962fef7ef555d2d]), partition=
I20240503 14:13:33.968094 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb79d46e3614382a451c6c021a8d997 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f2259e021ef4a648962fef7ef555d2d]), partition=
I20240503 14:13:33.968528 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb79d46e3614382a451c6c021a8d997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.969449 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4363edcc954769a3ab9c658e198c2d with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:33.970486 6653 catalog_manager.cc:5484] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:33.978058 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb79d46e3614382a451c6c021a8d997 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3f2259e021ef4a648962fef7ef555d2d]), partition=
I20240503 14:13:33.978325 11992 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:33.978539 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb79d46e3614382a451c6c021a8d997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.989137 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ff0991a4e245609405ff945388417b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:33.989835 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb79d46e3614382a451c6c021a8d997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.001928 9798 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:34.008832 12001 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:34.010947 12001 tablet_bootstrap.cc:654] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.014508 11985 ts_tablet_manager.cc:1888] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:34.016671 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b4363edcc954769a3ab9c658e198c2d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:34.017253 11985 log.cc:1196] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/f0611c8f61fe4ac2bb94818c7a804c97
I20240503 14:13:34.020658 11985 ts_tablet_manager.cc:1909] T f0611c8f61fe4ac2bb94818c7a804c97 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:34.023954 11992 tablet_bootstrap.cc:654] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.026494 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet f0611c8f61fe4ac2bb94818c7a804c97 (table rebalancer_dynamic_table_new_name [id=fe456347dbd74c3c809e0d38df0987d6]) successfully deleted
I20240503 14:13:34.028919 6930 raft_consensus.cc:1261] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:34.031148 9531 raft_consensus.cc:1261] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:34.034876 9662 raft_consensus.cc:1261] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:34.035634 11876 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240503 14:13:34.032907 12007 tablet_bootstrap.cc:492] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:34.042176 11874 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:34.044632 12003 tablet_replica.cc:330] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:34.044932 12003 raft_consensus.cc:2229] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:34.045241 12003 raft_consensus.cc:2258] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:34.045841 12007 tablet_bootstrap.cc:654] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:34.051096 9468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 83399us
I20240503 14:13:34.053913 11999 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:34.049381 12003 ts_tablet_manager.cc:1875] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:34.058033 7058 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:34.059674 12005 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:34.067237 11992 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:34.068020 11992 ts_tablet_manager.cc:1367] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.090s user 0.008s sys 0.000s
I20240503 14:13:34.069682 11992 raft_consensus.cc:348] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.073536 12006 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:34.075773 12007 tablet_bootstrap.cc:492] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:34.076005 12006 tablet_bootstrap.cc:654] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.076083 12007 ts_tablet_manager.cc:1367] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.000s
I20240503 14:13:34.077136 12007 raft_consensus.cc:348] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.077441 12007 raft_consensus.cc:374] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.077534 12007 raft_consensus.cc:729] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.077805 12007 consensus_queue.cc:260] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.078459 12007 ts_tablet_manager.cc:1398] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:34.081729 11990 tablet_bootstrap.cc:492] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:34.081975 11990 ts_tablet_manager.cc:1367] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.162s user 0.010s sys 0.045s
I20240503 14:13:34.082860 11990 raft_consensus.cc:348] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:34.083185 11990 raft_consensus.cc:374] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.083348 11990 raft_consensus.cc:729] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.083724 11990 consensus_queue.cc:260] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:34.084334 11990 ts_tablet_manager.cc:1398] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:34.084705 11990 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:34.085341 11992 raft_consensus.cc:374] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.085464 11992 raft_consensus.cc:729] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.079245 7058 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:34.088371 11990 tablet_bootstrap.cc:654] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.088959 11992 consensus_queue.cc:260] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.089751 11992 ts_tablet_manager.cc:1398] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 14:13:34.092299 9530 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:34.101610 11990 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:34.102358 11990 ts_tablet_manager.cc:1367] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240503 14:13:34.102228 6654 catalog_manager.cc:5484] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:34.103473 11876 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:34.104043 11990 raft_consensus.cc:348] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.104974 11990 raft_consensus.cc:374] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.105249 11990 raft_consensus.cc:729] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.105743 11990 consensus_queue.cc:260] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.106974 11990 ts_tablet_manager.cc:1398] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:34.114368 11988 tablet_bootstrap.cc:654] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.119827 6930 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:34.120446 11989 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:34.120638 11989 ts_tablet_manager.cc:1367] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.184s user 0.023s sys 0.042s
I20240503 14:13:34.121511 11989 raft_consensus.cc:348] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.121788 11989 raft_consensus.cc:374] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.121870 11989 raft_consensus.cc:729] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.122112 11989 consensus_queue.cc:260] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.122648 11989 ts_tablet_manager.cc:1398] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:34.122843 12003 ts_tablet_manager.cc:1888] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:34.122961 11989 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:34.122968 12003 log.cc:1196] T 4b4363edcc954769a3ab9c658e198c2d P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/4b4363edcc954769a3ab9c658e198c2d
I20240503 14:13:34.123921 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 4b4363edcc954769a3ab9c658e198c2d (table rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]) successfully deleted
I20240503 14:13:34.136091 11989 tablet_bootstrap.cc:654] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.142467 9798 consensus_queue.cc:237] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:34.144858 11977 raft_consensus.cc:482] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.145015 11977 raft_consensus.cc:504] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:34.145368 11989 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:34.145517 11989 ts_tablet_manager.cc:1367] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20240503 14:13:34.145745 11977 leader_election.cc:288] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:34.146342 11989 raft_consensus.cc:348] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.146852 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df05d420e5cd426e88cd33de16a707f6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:34.147270 6930 raft_consensus.cc:2454] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:34.148326 9863 leader_election.cc:302] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:34.149920 11977 raft_consensus.cc:2781] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.150045 11977 raft_consensus.cc:482] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.150138 11977 raft_consensus.cc:3037] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.151157 12006 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:34.151348 12006 ts_tablet_manager.cc:1367] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.000s
I20240503 14:13:34.152458 11988 tablet_bootstrap.cc:492] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:34.152618 11988 ts_tablet_manager.cc:1367] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.269s user 0.002s sys 0.003s
I20240503 14:13:34.152482 12006 raft_consensus.cc:348] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.152760 12006 raft_consensus.cc:374] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.152868 12006 raft_consensus.cc:729] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.153219 12006 consensus_queue.cc:260] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.153210 11977 raft_consensus.cc:504] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:34.154457 11988 raft_consensus.cc:348] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:34.154781 11988 raft_consensus.cc:374] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.154906 11988 raft_consensus.cc:729] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.155000 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df05d420e5cd426e88cd33de16a707f6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:34.155164 11988 consensus_queue.cc:260] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:34.155359 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df05d420e5cd426e88cd33de16a707f6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:34.155445 6929 raft_consensus.cc:3037] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.155658 9663 raft_consensus.cc:3037] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.155673 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df05d420e5cd426e88cd33de16a707f6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:34.156785 6930 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:34.157120 11989 raft_consensus.cc:374] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.157236 11989 raft_consensus.cc:729] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.157415 11775 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:34.157958 12001 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:34.158093 12001 ts_tablet_manager.cc:1367] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.149s user 0.005s sys 0.006s
I20240503 14:13:34.158468 11989 consensus_queue.cc:260] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.160171 11225 raft_consensus.cc:482] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.160120 12001 raft_consensus.cc:348] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.160318 12001 raft_consensus.cc:374] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.160404 12001 raft_consensus.cc:729] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.160320 11225 raft_consensus.cc:504] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.160647 12001 consensus_queue.cc:260] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.161093 11225 leader_election.cc:288] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:34.161245 12001 ts_tablet_manager.cc:1398] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:34.161561 12001 tablet_bootstrap.cc:492] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:34.161671 9663 raft_consensus.cc:2454] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:34.162081 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff0991a4e245609405ff945388417b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:34.162304 7058 raft_consensus.cc:2454] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:34.162585 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff0991a4e245609405ff945388417b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:34.162815 9929 raft_consensus.cc:2454] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:34.162842 9735 leader_election.cc:302] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 3efdbd3506274dba846ae40404f0572c; no voters:
I20240503 14:13:34.161363 6929 raft_consensus.cc:2454] T df05d420e5cd426e88cd33de16a707f6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:34.163115 11225 raft_consensus.cc:2781] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.163214 12001 tablet_bootstrap.cc:654] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.163229 11225 raft_consensus.cc:482] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.163322 11225 raft_consensus.cc:3037] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.163903 11988 ts_tablet_manager.cc:1398] T df05d420e5cd426e88cd33de16a707f6 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240503 14:13:34.165424 11989 ts_tablet_manager.cc:1398] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20240503 14:13:34.165988 9863 leader_election.cc:302] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:34.166353 11225 raft_consensus.cc:504] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.166712 11976 raft_consensus.cc:2781] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.166888 11976 raft_consensus.cc:686] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:34.167356 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff0991a4e245609405ff945388417b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:34.167284 11976 consensus_queue.cc:237] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:34.167547 7058 raft_consensus.cc:3037] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.170321 7058 raft_consensus.cc:2454] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:34.171133 9735 leader_election.cc:302] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 3efdbd3506274dba846ae40404f0572c; no voters:
I20240503 14:13:34.171486 11664 raft_consensus.cc:2781] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.171649 11225 leader_election.cc:288] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:34.171671 11664 raft_consensus.cc:686] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:34.171931 11664 consensus_queue.cc:237] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:34.172976 12006 ts_tablet_manager.cc:1398] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.022s user 0.000s sys 0.003s
I20240503 14:13:34.173244 12006 tablet_bootstrap.cc:492] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:34.177304 12006 tablet_bootstrap.cc:654] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:34.179098 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89ff0991a4e245609405ff945388417b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:34.179455 9929 raft_consensus.cc:3037] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.186008 12006 tablet_bootstrap.cc:492] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:34.186591 12006 ts_tablet_manager.cc:1367] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.003s
I20240503 14:13:34.187567 12006 raft_consensus.cc:348] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.188671 12006 raft_consensus.cc:374] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.188814 12006 raft_consensus.cc:729] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.189224 12006 consensus_queue.cc:260] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.190076 12006 ts_tablet_manager.cc:1398] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:34.195854 7188 consensus_queue.cc:237] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:34.196015 9929 raft_consensus.cc:2454] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:34.199386 9530 raft_consensus.cc:1261] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:34.199651 9734 consensus_peers.cc:485] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7760fdb06dac4a46ab4452a67d7110aa. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:34.200119 11816 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:34.200428 7058 raft_consensus.cc:1261] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:34.201265 11816 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:34.201808 6929 raft_consensus.cc:1261] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:34.202340 11878 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:34.205185 11999 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:34.208443 6930 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:34.201882 11977 leader_election.cc:288] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:34.212913 9662 raft_consensus.cc:1261] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:34.213680 11816 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:34.214119 11872 consensus_queue.cc:1035] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:34.215904 9662 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:34.217150 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e5410b22c57427d96a32fc9957cea8e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:34.217831 9531 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:34.219765 6658 catalog_manager.cc:5484] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:34.224120 6653 catalog_manager.cc:5484] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:34.224715 12001 tablet_bootstrap.cc:492] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:34.225121 12001 ts_tablet_manager.cc:1367] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.064s user 0.013s sys 0.023s
I20240503 14:13:34.225911 12001 raft_consensus.cc:348] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.227140 12001 raft_consensus.cc:374] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:34.227274 12001 raft_consensus.cc:729] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:34.227576 12001 consensus_queue.cc:260] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.232638 7060 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:34.237141 12001 ts_tablet_manager.cc:1398] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240503 14:13:34.237654 12025 ts_tablet_manager.cc:902] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:34.240075 9530 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:34.244107 11775 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:34.249540 6653 catalog_manager.cc:5484] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:34.256894 6654 catalog_manager.cc:5484] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:34.261979 11806 raft_consensus.cc:482] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.262187 11806 raft_consensus.cc:504] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.262869 11806 leader_election.cc:288] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:34.263454 7188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb79d46e3614382a451c6c021a8d997" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:34.263764 7188 raft_consensus.cc:2454] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:34.264642 9468 leader_election.cc:302] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:34.265072 11806 raft_consensus.cc:2781] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.265036 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb79d46e3614382a451c6c021a8d997" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:34.265192 11806 raft_consensus.cc:482] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.265278 9798 raft_consensus.cc:2454] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:34.265314 11806 raft_consensus.cc:3037] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.267241 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e5410b22c57427d96a32fc9957cea8e with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:34.267647 11984 tablet_replica.cc:330] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:34.267951 11984 raft_consensus.cc:2229] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:34.268219 11984 raft_consensus.cc:2258] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:34.269486 11984 ts_tablet_manager.cc:1875] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:34.272477 11806 raft_consensus.cc:504] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.274102 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb79d46e3614382a451c6c021a8d997" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:34.274420 9798 raft_consensus.cc:3037] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.274621 11806 leader_election.cc:288] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:34.274718 7188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb79d46e3614382a451c6c021a8d997" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:34.274920 7188 raft_consensus.cc:3037] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.278368 9662 raft_consensus.cc:2932] T 7e5410b22c57427d96a32fc9957cea8e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:34.279553 7188 raft_consensus.cc:2454] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:34.280205 9468 leader_election.cc:302] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:34.280568 11806 raft_consensus.cc:2781] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.280730 11806 raft_consensus.cc:686] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:34.281081 11806 consensus_queue.cc:237] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:34.285144 9798 raft_consensus.cc:2454] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:34.290207 11978 raft_consensus.cc:1050] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
I20240503 14:13:34.293166 7188 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:34.293937 11775 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:34.291540 11978 consensus_queue.cc:237] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:34.312022 6651 catalog_manager.cc:5484] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:34.318358 11984 ts_tablet_manager.cc:1888] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:34.318691 11984 log.cc:1196] T 7e5410b22c57427d96a32fc9957cea8e P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/7e5410b22c57427d96a32fc9957cea8e
I20240503 14:13:34.319897 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 7e5410b22c57427d96a32fc9957cea8e (table rebalancer_extra_table_4 [id=6d75467edf5a417cbfeb46b1b81f1f2e]) successfully deleted
I20240503 14:13:34.322820 7191 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:34.323661 11977 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:34.325933 9664 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:34.327239 11977 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:34.333992 9531 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:34.334885 11977 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:34.345715 11400 raft_consensus.cc:1050] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:34.346756 11400 consensus_queue.cc:237] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:34.354799 7060 consensus_queue.cc:237] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:34.357932 12045 ts_tablet_manager.cc:902] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:34.362179 9532 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:34.364385 10371 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:34.366932 9530 raft_consensus.cc:1261] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:34.367961 11739 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:34.369110 7060 raft_consensus.cc:1261] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:34.370169 6931 raft_consensus.cc:1261] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:34.370805 11739 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:34.371307 11739 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:34.402904 11976 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:34.406263 10371 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.405743 12045 tablet_copy_client.cc:308] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.5:36609
I20240503 14:13:34.408839 9553 tablet_copy_service.cc:140] P 5b7e36bf354f480c90859cb8867b59cf: Received BeginTabletCopySession request for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:60715)
I20240503 14:13:34.409018 9553 tablet_copy_service.cc:161] P 5b7e36bf354f480c90859cb8867b59cf: Beginning new tablet copy session on tablet d9641f7f5a3e4debb5aabc6b8a7497f6 from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:60715: session id = ea801601713a4d55aee0de9e3473e0bc-d9641f7f5a3e4debb5aabc6b8a7497f6
I20240503 14:13:34.409013 9664 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:34.410821 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d3ec4f0833d4d489f774757312efb32 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:34.410593 9533 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.412307 9553 tablet_copy_source_session.cc:205] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:34.413785 12045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9641f7f5a3e4debb5aabc6b8a7497f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.415040 6651 catalog_manager.cc:5484] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 7, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 2 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:34.421416 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3ec4f0833d4d489f774757312efb32 with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:34.422196 7190 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:34.420861 6653 catalog_manager.cc:5484] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:34.428095 6929 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:34.429282 11775 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:34.431027 6929 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.433961 11806 raft_consensus.cc:482] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:34.434134 11806 raft_consensus.cc:504] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.435068 11806 leader_election.cc:288] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:34.435581 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b86d126b92400f9b14a9e4fb527fc9" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:34.435856 6928 raft_consensus.cc:2454] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:34.436681 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b86d126b92400f9b14a9e4fb527fc9" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:34.436882 9929 raft_consensus.cc:2454] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:34.437583 9467 leader_election.cc:302] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:34.438441 11806 raft_consensus.cc:2781] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:34.439136 11806 raft_consensus.cc:482] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:34.439236 11806 raft_consensus.cc:3037] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.442085 11739 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.442647 11806 raft_consensus.cc:504] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.442763 6929 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.443856 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b86d126b92400f9b14a9e4fb527fc9" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:34.444124 6928 raft_consensus.cc:3037] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.447225 6928 raft_consensus.cc:2454] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:34.448211 9530 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.449522 9467 leader_election.cc:302] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:34.449929 11996 raft_consensus.cc:2781] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:34.450080 11996 raft_consensus.cc:686] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:34.450516 11996 consensus_queue.cc:237] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:34.456732 6653 catalog_manager.cc:5484] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:34.461952 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4b86d126b92400f9b14a9e4fb527fc9" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:34.462224 9929 raft_consensus.cc:3037] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:34.464984 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:34.468019 9929 raft_consensus.cc:2454] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:34.473906 9798 raft_consensus.cc:1261] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:34.474532 12002 consensus_queue.cc:1035] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.474802 7058 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.476615 7190 raft_consensus.cc:1261] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:34.486286 6800 consensus_queue.cc:237] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:34.488366 12002 consensus_queue.cc:1035] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.488546 11806 leader_election.cc:288] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:34.490773 7060 raft_consensus.cc:1261] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:34.491217 9530 raft_consensus.cc:1261] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:34.491392 11739 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:34.492578 11739 consensus_queue.cc:1035] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:34.495246 11739 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.496836 9530 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.513756 12045 tablet_copy_client.cc:783] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:34.515554 7061 raft_consensus.cc:2932] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.524792 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dcb23bc5b0a44da7989aa828826ece07 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:34.526213 6653 catalog_manager.cc:5484] T dcb23bc5b0a44da7989aa828826ece07 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 5 to 6, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:34.532826 12045 tablet_copy_client.cc:647] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:34.535682 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet dcb23bc5b0a44da7989aa828826ece07 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:34.548246 12076 tablet_replica.cc:330] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:34.549527 12076 raft_consensus.cc:2229] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:34.549922 12076 raft_consensus.cc:2258] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:34.551169 12076 ts_tablet_manager.cc:1875] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:34.553100 6655 catalog_manager.cc:5484] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:34.560067 9530 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:34.562872 6654 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 9bb79d46e3614382a451c6c021a8d997 (table rebalancer_dynamic_table [id=3f2259e021ef4a648962fef7ef555d2d]). Expected version 1 got 0
I20240503 14:13:34.565761 12035 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.582384 6655 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 9bb79d46e3614382a451c6c021a8d997 (table rebalancer_dynamic_table [id=3f2259e021ef4a648962fef7ef555d2d]). Expected version 1 got 0
I20240503 14:13:34.575536 12077 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.585233 11976 consensus_queue.cc:1035] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.586850 12077 tablet.cc:1620] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:34.576050 12078 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.593291 12078 tablet.cc:1620] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:34.593906 12045 tablet_copy_client.cc:523] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:34.597841 12064 tablet_replica.cc:330] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:34.598146 12064 raft_consensus.cc:2229] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:34.598520 12064 pending_rounds.cc:62] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Trying to abort 1 pending ops.
I20240503 14:13:34.598699 12064 pending_rounds.cc:68] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Aborting op as it isn't in flight: id { term: 2 index: 6 } timestamp: 7023598033814171648 op_type: NO_OP noop_request { }
I20240503 14:13:34.598950 12064 raft_consensus.cc:2866] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:13:34.599117 12064 raft_consensus.cc:2258] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:34.606956 12064 ts_tablet_manager.cc:1875] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:34.608305 12080 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:34.609078 12080 tablet.cc:1620] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:34.609687 12076 ts_tablet_manager.cc:1888] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:34.610455 12076 log.cc:1196] T dcb23bc5b0a44da7989aa828826ece07 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/dcb23bc5b0a44da7989aa828826ece07
I20240503 14:13:34.612401 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet dcb23bc5b0a44da7989aa828826ece07 (table rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]) successfully deleted
I20240503 14:13:34.613832 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=3f2259e021ef4a648962fef7ef555d2d] alter complete (version 1)
I20240503 14:13:34.619081 12025 tablet_copy_client.cc:308] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:34.620344 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 9b154f780fc742fa9b0efd609c307f95 from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:60021)
I20240503 14:13:34.620577 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 9b154f780fc742fa9b0efd609c307f95 from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:60021: session id = f03dc37fe3234033bc665bed4f1b51f5-9b154f780fc742fa9b0efd609c307f95
I20240503 14:13:34.626479 6951 tablet_copy_source_session.cc:205] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:34.628063 11225 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.628926 6650 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 9bb79d46e3614382a451c6c021a8d997 (table rebalancer_dynamic_table [id=3f2259e021ef4a648962fef7ef555d2d]). Expected version 1 got 0
I20240503 14:13:34.631651 11977 consensus_queue.cc:1035] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:34.643702 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b154f780fc742fa9b0efd609c307f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.644388 12045 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:34.664640 12064 ts_tablet_manager.cc:1888] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 14:13:34.668408 12025 tablet_copy_client.cc:783] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:34.668733 12025 tablet_copy_client.cc:647] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:34.672933 12064 log.cc:1196] T 1d3ec4f0833d4d489f774757312efb32 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/1d3ec4f0833d4d489f774757312efb32
I20240503 14:13:34.674310 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 1d3ec4f0833d4d489f774757312efb32 (table rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]) successfully deleted
I20240503 14:13:34.675226 12025 tablet_copy_client.cc:523] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:34.683974 12045 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:34.684504 12045 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:34.685228 12025 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:34.686538 12045 ts_tablet_manager.cc:1367] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.009s
I20240503 14:13:34.690886 12045 raft_consensus.cc:348] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:34.691138 12045 raft_consensus.cc:729] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:34.691340 12045 consensus_queue.cc:260] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:34.692157 12045 ts_tablet_manager.cc:1398] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:34.702251 9553 tablet_copy_service.cc:342] P 5b7e36bf354f480c90859cb8867b59cf: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-d9641f7f5a3e4debb5aabc6b8a7497f6 received from {username='slave'} at 127.0.186.8:60715
I20240503 14:13:34.702414 9553 tablet_copy_service.cc:433] P 5b7e36bf354f480c90859cb8867b59cf: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-d9641f7f5a3e4debb5aabc6b8a7497f6 on tablet d9641f7f5a3e4debb5aabc6b8a7497f6 with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:34.724390 12090 ts_tablet_manager.cc:902] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:34.739677 6655 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:34.750861 12077 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:34.751349 12077 tablet.cc:1620] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:34.758939 12080 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:34.759478 12080 tablet.cc:1620] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:34.773921 12078 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:34.774542 12078 tablet.cc:1620] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:34.775161 9929 raft_consensus.cc:1203] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:34.780342 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=3f2259e021ef4a648962fef7ef555d2d] alter complete (version 2)
I20240503 14:13:34.839469 12090 tablet_copy_client.cc:308] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:34.840795 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 7760fdb06dac4a46ab4452a67d7110aa from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:58411)
I20240503 14:13:34.841014 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 7760fdb06dac4a46ab4452a67d7110aa from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:58411: session id = e1bf5fb7402c46829974e2f74cd88708-7760fdb06dac4a46ab4452a67d7110aa
I20240503 14:13:34.844269 9819 tablet_copy_source_session.cc:205] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:34.846086 12090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7760fdb06dac4a46ab4452a67d7110aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:34.856886 12025 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:34.857403 12025 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:34.857714 12025 ts_tablet_manager.cc:1367] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.173s user 0.056s sys 0.009s
I20240503 14:13:34.858608 12025 raft_consensus.cc:348] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:34.858891 12025 raft_consensus.cc:729] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:34.859122 12025 consensus_queue.cc:260] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:34.865208 12090 tablet_copy_client.cc:783] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:34.865492 12090 tablet_copy_client.cc:647] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:34.866119 12025 ts_tablet_manager.cc:1398] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:13:34.867301 12090 tablet_copy_client.cc:523] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:34.867762 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-9b154f780fc742fa9b0efd609c307f95 received from {username='slave'} at 127.0.186.6:60021
I20240503 14:13:34.867882 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-9b154f780fc742fa9b0efd609c307f95 on tablet 9b154f780fc742fa9b0efd609c307f95 with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:34.872820 12090 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:34.880824 12090 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:34.881422 12090 tablet_bootstrap.cc:492] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:34.881747 12090 ts_tablet_manager.cc:1367] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20240503 14:13:34.883248 6653 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:34.883232 12090 raft_consensus.cc:348] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:34.883528 12090 raft_consensus.cc:729] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:34.883816 12090 consensus_queue.cc:260] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:34.892549 12090 ts_tablet_manager.cc:1398] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:13:34.901502 12077 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:34.903055 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-7760fdb06dac4a46ab4452a67d7110aa received from {username='slave'} at 127.0.186.4:58411
I20240503 14:13:34.903225 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-7760fdb06dac4a46ab4452a67d7110aa on tablet 7760fdb06dac4a46ab4452a67d7110aa with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:34.908248 12080 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:34.908828 12078 tablet.cc:1722] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:34.918010 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=3f2259e021ef4a648962fef7ef555d2d] alter complete (version 3)
I20240503 14:13:34.959261 9929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41194
I20240503 14:13:34.959517 9929 raft_consensus.cc:595] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:35.004930 6798 consensus_queue.cc:237] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:35.007316 7060 raft_consensus.cc:1261] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:35.007380 6929 raft_consensus.cc:1261] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:35.007561 6734 consensus_peers.cc:485] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34773b2ab9194e70b3e49357a2da4d8c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:35.008515 12066 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.008921 11400 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.011881 11400 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:35.012444 6929 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:35.013553 7060 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:35.019022 6653 catalog_manager.cc:5484] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 4, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:35.024583 6655 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:35.024863 6655 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:35.028955 6655 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 9bb79d46e3614382a451c6c021a8d997
I20240503 14:13:35.029999 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb79d46e3614382a451c6c021a8d997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:35 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:35.030297 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb79d46e3614382a451c6c021a8d997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:35 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:35.030514 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb79d46e3614382a451c6c021a8d997 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:35 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:35.031193 12153 tablet_replica.cc:330] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:35.031250 12064 tablet_replica.cc:330] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:35.031476 12153 raft_consensus.cc:2229] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:35.031488 12064 raft_consensus.cc:2229] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:35.031709 12064 raft_consensus.cc:2258] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:35.031790 12153 raft_consensus.cc:2258] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:35.032920 12064 ts_tablet_manager.cc:1875] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:35.038586 12153 ts_tablet_manager.cc:1875] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:35.041800 11996 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.045536 12064 ts_tablet_manager.cc:1888] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:35.045907 12064 log.cc:1196] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/9bb79d46e3614382a451c6c021a8d997
I20240503 14:13:35.046540 12064 ts_tablet_manager.cc:1909] T 9bb79d46e3614382a451c6c021a8d997 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:35.047849 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 9bb79d46e3614382a451c6c021a8d997 (table rebalancer_dynamic_table_new_name [id=3f2259e021ef4a648962fef7ef555d2d]) successfully deleted
I20240503 14:13:35.052742 12152 tablet_replica.cc:330] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:35.053131 12152 raft_consensus.cc:2229] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:35.053566 12152 raft_consensus.cc:2258] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:35.055765 12153 ts_tablet_manager.cc:1888] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:35.055932 12153 log.cc:1196] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/9bb79d46e3614382a451c6c021a8d997
I20240503 14:13:35.056397 12153 ts_tablet_manager.cc:1909] T 9bb79d46e3614382a451c6c021a8d997 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:35.057566 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 9bb79d46e3614382a451c6c021a8d997 (table rebalancer_dynamic_table_new_name [id=3f2259e021ef4a648962fef7ef555d2d]) successfully deleted
I20240503 14:13:35.058815 12152 ts_tablet_manager.cc:1875] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:35.062336 12002 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.065601 12152 ts_tablet_manager.cc:1888] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:35.065771 12152 log.cc:1196] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/9bb79d46e3614382a451c6c021a8d997
I20240503 14:13:35.066426 12152 ts_tablet_manager.cc:1909] T 9bb79d46e3614382a451c6c021a8d997 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:35.067960 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 9bb79d46e3614382a451c6c021a8d997 (table rebalancer_dynamic_table_new_name [id=3f2259e021ef4a648962fef7ef555d2d]) successfully deleted
I20240503 14:13:35.079850 7191 raft_consensus.cc:1203] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:35.081709 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:35.095484 9779 tablet_service.cc:1467] Processing CreateTablet for tablet ef4166613a824863b6c006dafb3783d3 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.095615 7170 tablet_service.cc:1467] Processing CreateTablet for tablet ef4166613a824863b6c006dafb3783d3 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.096041 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4166613a824863b6c006dafb3783d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.096112 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4166613a824863b6c006dafb3783d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.097489 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.097595 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.098028 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.098039 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.108449 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 005dec40a9054f7fa95ea8722e754e24 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.109009 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005dec40a9054f7fa95ea8722e754e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.111168 12171 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:35.112105 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 005dec40a9054f7fa95ea8722e754e24 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.112658 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005dec40a9054f7fa95ea8722e754e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.113678 12171 tablet_bootstrap.cc:654] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.118849 12171 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:35.119117 12171 ts_tablet_manager.cc:1367] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240503 14:13:35.120131 12171 raft_consensus.cc:348] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.120501 12171 raft_consensus.cc:374] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.120673 12171 raft_consensus.cc:729] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.125783 12172 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:35.130549 12171 consensus_queue.cc:260] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.130867 9511 tablet_service.cc:1467] Processing CreateTablet for tablet ef4166613a824863b6c006dafb3783d3 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.131405 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4166613a824863b6c006dafb3783d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.132234 12171 ts_tablet_manager.cc:1398] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:35.134752 12170 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:35.135928 12171 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:35.136808 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:35.139024 12170 tablet_bootstrap.cc:654] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.139855 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.140331 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.143704 11977 raft_consensus.cc:979] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: : Instructing follower 5b7e36bf354f480c90859cb8867b59cf to start an election
I20240503 14:13:35.143880 11977 raft_consensus.cc:1067] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Signalling peer 5b7e36bf354f480c90859cb8867b59cf to start an election
I20240503 14:13:35.143972 12171 tablet_bootstrap.cc:654] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.145207 9530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964"
dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
from {username='slave'} at 127.0.186.8:60715
I20240503 14:13:35.145457 9530 raft_consensus.cc:482] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:35.145568 9530 raft_consensus.cc:3037] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:35.147243 12170 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:35.148626 12170 ts_tablet_manager.cc:1367] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20240503 14:13:35.150080 12170 raft_consensus.cc:348] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.151182 12170 raft_consensus.cc:374] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.151466 12170 raft_consensus.cc:729] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.152132 12170 consensus_queue.cc:260] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.153424 7040 tablet_service.cc:1467] Processing CreateTablet for tablet af479b9027c84a1680b30ccf44b464d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c]), partition=
I20240503 14:13:35.153885 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af479b9027c84a1680b30ccf44b464d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.154006 12170 ts_tablet_manager.cc:1398] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:35.158138 12170 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:35.159042 12171 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:35.159243 12171 ts_tablet_manager.cc:1367] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240503 14:13:35.160362 12171 raft_consensus.cc:348] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.160614 12171 raft_consensus.cc:374] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.160698 12171 raft_consensus.cc:729] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.160921 12171 consensus_queue.cc:260] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.162209 9779 tablet_service.cc:1467] Processing CreateTablet for tablet af479b9027c84a1680b30ccf44b464d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c]), partition=
I20240503 14:13:35.162719 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af479b9027c84a1680b30ccf44b464d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.166247 12170 tablet_bootstrap.cc:654] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.167028 12171 ts_tablet_manager.cc:1398] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 14:13:35.167735 11806 raft_consensus.cc:1050] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:35.168610 11806 consensus_queue.cc:237] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:35.150149 9530 raft_consensus.cc:504] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:35.176154 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:35.176383 9929 raft_consensus.cc:3032] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:35.176487 9929 raft_consensus.cc:729] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:35.176792 9929 consensus_queue.cc:260] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:35.177284 9929 raft_consensus.cc:3037] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:35.181063 9929 raft_consensus.cc:2454] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:35.183084 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:35.183302 9664 raft_consensus.cc:3037] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:35.186245 9513 tablet_service.cc:1467] Processing CreateTablet for tablet af479b9027c84a1680b30ccf44b464d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c]), partition=
I20240503 14:13:35.186347 9664 raft_consensus.cc:2454] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:35.186730 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af479b9027c84a1680b30ccf44b464d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.192319 9467 leader_election.cc:302] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:35.192713 11996 raft_consensus.cc:2781] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:35.192844 11996 raft_consensus.cc:686] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:35.193120 11996 consensus_queue.cc:237] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:35.194584 12035 raft_consensus.cc:482] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.194753 12035 raft_consensus.cc:504] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.195302 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 005dec40a9054f7fa95ea8722e754e24 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:35.195485 12035 leader_election.cc:288] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:35.196267 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad2f2ef32d4fc88cdd62bfeab97b8a" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:35.196388 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad2f2ef32d4fc88cdd62bfeab97b8a" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
W20240503 14:13:35.197022 9735 leader_election.cc:341] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
W20240503 14:13:35.197190 9733 leader_election.cc:341] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:35.197546 9733 leader_election.cc:302] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:35.198050 12035 raft_consensus.cc:2726] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:35.198083 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d91b6f4bb244c78bc6fef8b837f964" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:35.198318 7191 raft_consensus.cc:3037] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:35.198908 6653 catalog_manager.cc:5484] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 1 to 2, leader changed from ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.201314 7191 raft_consensus.cc:2454] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:35.202244 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005dec40a9054f7fa95ea8722e754e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.203554 9798 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:35.203667 7060 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:35.204468 11996 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:35.204864 12015 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:35.205495 9929 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:35.206048 12015 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:35.207847 12171 tablet_bootstrap.cc:492] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:35.206053 9530 leader_election.cc:288] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:35.209441 12171 tablet_bootstrap.cc:654] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.217950 12180 tablet_bootstrap.cc:492] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:35.224730 9664 raft_consensus.cc:1203] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:35.231245 12171 tablet_bootstrap.cc:492] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:35.231436 12171 ts_tablet_manager.cc:1367] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.007s
I20240503 14:13:35.232264 12171 raft_consensus.cc:348] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.234735 12171 raft_consensus.cc:374] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.234958 12171 raft_consensus.cc:729] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.235272 12171 consensus_queue.cc:260] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.243710 12171 ts_tablet_manager.cc:1398] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240503 14:13:35.243834 11996 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:35.244741 12174 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:35.246443 12172 tablet_bootstrap.cc:700] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: Time spent opening tablet: real 0.117s user 0.001s sys 0.000s
I20240503 14:13:35.246927 12172 tablet_bootstrap.cc:654] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.249809 7060 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:35.251189 9798 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:35.253755 12174 tablet_bootstrap.cc:654] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.253975 12180 tablet_bootstrap.cc:654] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.254034 6655 catalog_manager.cc:5484] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 2 to 3, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.259133 9928 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:35.260794 11225 raft_consensus.cc:482] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.260972 11225 raft_consensus.cc:504] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.261660 11225 leader_election.cc:288] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:35.262429 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4166613a824863b6c006dafb3783d3" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:35.262689 7191 raft_consensus.cc:2454] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:35.263501 9734 leader_election.cc:302] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:35.263655 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4166613a824863b6c006dafb3783d3" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
W20240503 14:13:35.264314 9735 leader_election.cc:341] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:35.264678 11225 raft_consensus.cc:2781] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.264771 11225 raft_consensus.cc:482] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.264876 11225 raft_consensus.cc:3037] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.269202 12170 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:35.269418 12170 ts_tablet_manager.cc:1367] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.111s user 0.007s sys 0.004s
I20240503 14:13:35.269381 11225 raft_consensus.cc:504] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.270529 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4166613a824863b6c006dafb3783d3" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:35.270751 7191 raft_consensus.cc:3037] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.270648 12170 raft_consensus.cc:348] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.270891 12170 raft_consensus.cc:374] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.270973 12170 raft_consensus.cc:729] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.271263 12170 consensus_queue.cc:260] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.272243 12170 ts_tablet_manager.cc:1398] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:35.273725 7191 raft_consensus.cc:2454] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:35.274417 9734 leader_election.cc:302] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:35.274767 11816 raft_consensus.cc:2781] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.274909 11816 raft_consensus.cc:686] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:35.275158 11816 consensus_queue.cc:237] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.276361 12172 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:35.276541 12172 ts_tablet_manager.cc:1367] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.151s user 0.006s sys 0.000s
I20240503 14:13:35.277304 12172 raft_consensus.cc:348] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.278537 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4166613a824863b6c006dafb3783d3" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
W20240503 14:13:35.279107 9735 leader_election.cc:341] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:35.279538 11225 leader_election.cc:288] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:35.285174 12172 raft_consensus.cc:374] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.285348 12172 raft_consensus.cc:729] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.284988 12174 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:35.291075 12174 ts_tablet_manager.cc:1367] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.046s user 0.006s sys 0.000s
I20240503 14:13:35.290719 12172 consensus_queue.cc:260] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.293982 12172 ts_tablet_manager.cc:1398] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.017s user 0.003s sys 0.000s
I20240503 14:13:35.297014 9530 consensus_queue.cc:237] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:35.298099 12172 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:35.299921 9928 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:35.300484 11996 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.304158 11996 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:35.308576 9930 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:35.317411 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:35.319175 6653 catalog_manager.cc:5484] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:35.298491 12174 raft_consensus.cc:348] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.322743 12174 raft_consensus.cc:374] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.323050 12174 raft_consensus.cc:729] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.323958 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:35.330196 9795 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:35.336488 6655 catalog_manager.cc:5484] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.330989 11996 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.340368 12174 consensus_queue.cc:260] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.341123 12180 tablet_bootstrap.cc:492] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:35.341740 12172 tablet_bootstrap.cc:654] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.342113 12180 ts_tablet_manager.cc:1367] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.124s user 0.007s sys 0.003s
I20240503 14:13:35.341645 9798 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:35.342818 12193 tablet_replica.cc:330] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:35.345127 12193 raft_consensus.cc:2229] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:35.348659 12180 raft_consensus.cc:348] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.351420 12193 raft_consensus.cc:2258] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:35.351680 12180 raft_consensus.cc:374] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.352164 12180 raft_consensus.cc:729] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.352614 12180 consensus_queue.cc:260] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.353215 12174 ts_tablet_manager.cc:1398] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.061s user 0.002s sys 0.000s
I20240503 14:13:35.353324 12193 ts_tablet_manager.cc:1875] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:35.355008 12180 ts_tablet_manager.cc:1398] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:35.358338 12174 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:35.366015 12174 tablet_bootstrap.cc:654] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.372277 12193 ts_tablet_manager.cc:1888] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:35.372556 12193 log.cc:1196] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/d9641f7f5a3e4debb5aabc6b8a7497f6
I20240503 14:13:35.373437 12172 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:35.373674 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet d9641f7f5a3e4debb5aabc6b8a7497f6 (table rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]) successfully deleted
I20240503 14:13:35.373737 12172 ts_tablet_manager.cc:1367] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20240503 14:13:35.374742 12172 raft_consensus.cc:348] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.375066 12172 raft_consensus.cc:374] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.375182 12172 raft_consensus.cc:729] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.375442 12172 consensus_queue.cc:260] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.376076 12172 ts_tablet_manager.cc:1398] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:35.377307 12174 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:35.377511 12174 ts_tablet_manager.cc:1367] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240503 14:13:35.378201 12174 raft_consensus.cc:348] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.378412 12174 raft_consensus.cc:374] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.378504 12174 raft_consensus.cc:729] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.378872 12174 consensus_queue.cc:260] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.384224 12174 ts_tablet_manager.cc:1398] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:35.389988 12174 tablet_bootstrap.cc:492] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:35.400460 12174 tablet_bootstrap.cc:654] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.408284 12200 ts_tablet_manager.cc:902] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:35.410676 12174 tablet_bootstrap.cc:492] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:35.411029 12174 ts_tablet_manager.cc:1367] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.006s
I20240503 14:13:35.412187 12174 raft_consensus.cc:348] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.412618 12174 raft_consensus.cc:374] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.412782 12174 raft_consensus.cc:729] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.413199 12174 consensus_queue.cc:260] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.413925 12174 ts_tablet_manager.cc:1398] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:35.414549 12174 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:35.417457 12174 tablet_bootstrap.cc:654] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:35.427266 11225 raft_consensus.cc:482] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.427536 11225 raft_consensus.cc:504] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.428439 11225 leader_election.cc:288] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:35.429399 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af479b9027c84a1680b30ccf44b464d1" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:35.429663 7060 raft_consensus.cc:2454] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:35.429749 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af479b9027c84a1680b30ccf44b464d1" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:35.430003 9531 raft_consensus.cc:2454] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:35.430220 9735 leader_election.cc:302] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 3efdbd3506274dba846ae40404f0572c; no voters:
I20240503 14:13:35.430912 11816 raft_consensus.cc:2781] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.431021 11816 raft_consensus.cc:482] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.431129 11816 raft_consensus.cc:3037] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.437300 11816 raft_consensus.cc:504] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.438079 11816 leader_election.cc:288] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:35.439379 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af479b9027c84a1680b30ccf44b464d1" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:35.439702 9531 raft_consensus.cc:3037] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.442804 9531 raft_consensus.cc:2454] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:35.444548 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af479b9027c84a1680b30ccf44b464d1" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:35.444818 7060 raft_consensus.cc:3037] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.450829 9735 leader_election.cc:302] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:35.451623 12174 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:35.451826 12174 ts_tablet_manager.cc:1367] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.023s
I20240503 14:13:35.451562 11225 raft_consensus.cc:2781] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.452270 11225 raft_consensus.cc:686] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:35.452819 12174 raft_consensus.cc:348] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.452673 11225 consensus_queue.cc:237] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.452988 12174 raft_consensus.cc:374] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:35.453035 12174 raft_consensus.cc:729] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:35.453240 12174 consensus_queue.cc:260] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.453843 12174 ts_tablet_manager.cc:1398] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:35.457626 7060 raft_consensus.cc:2454] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:35.468017 12200 tablet_copy_client.cc:308] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:34101
I20240503 14:13:35.469105 6820 tablet_copy_service.cc:140] P f1a669e28f014fe287932acf56bebc13: Received BeginTabletCopySession request for tablet 34773b2ab9194e70b3e49357a2da4d8c from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:50489)
I20240503 14:13:35.469360 6820 tablet_copy_service.cc:161] P f1a669e28f014fe287932acf56bebc13: Beginning new tablet copy session on tablet 34773b2ab9194e70b3e49357a2da4d8c from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:50489: session id = ea801601713a4d55aee0de9e3473e0bc-34773b2ab9194e70b3e49357a2da4d8c
I20240503 14:13:35.473181 6655 catalog_manager.cc:5484] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:35.480644 6820 tablet_copy_source_session.cc:205] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:35.481818 12200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34773b2ab9194e70b3e49357a2da4d8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:35.486068 11806 raft_consensus.cc:482] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.486235 11806 raft_consensus.cc:504] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.486553 12200 tablet_copy_client.cc:783] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:35.486793 12200 tablet_copy_client.cc:647] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:35.487093 11806 leader_election.cc:288] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:35.487669 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4166613a824863b6c006dafb3783d3" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:35.487872 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef4166613a824863b6c006dafb3783d3" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:35.488154 7191 raft_consensus.cc:2379] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b7e36bf354f480c90859cb8867b59cf in current term 1: Already voted for candidate 3efdbd3506274dba846ae40404f0572c in this term.
I20240503 14:13:35.488659 12200 tablet_copy_client.cc:523] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:35.488619 9468 leader_election.cc:302] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:35.488970 11806 raft_consensus.cc:3037] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.491760 11806 raft_consensus.cc:2726] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:35.492234 12200 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:35.529147 12200 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:35.529567 12200 tablet_bootstrap.cc:492] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:35.529883 12200 ts_tablet_manager.cc:1367] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.038s user 0.027s sys 0.009s
I20240503 14:13:35.530643 12200 raft_consensus.cc:348] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:35.530865 12200 raft_consensus.cc:729] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:35.531082 12200 consensus_queue.cc:260] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:35.531559 12200 ts_tablet_manager.cc:1398] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:35.531663 11806 raft_consensus.cc:482] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.531867 11806 raft_consensus.cc:504] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.532578 11806 leader_election.cc:288] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:35.533082 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad2f2ef32d4fc88cdd62bfeab97b8a" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:35.533339 9798 raft_consensus.cc:2454] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:35.533389 6820 tablet_copy_service.cc:342] P f1a669e28f014fe287932acf56bebc13: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-34773b2ab9194e70b3e49357a2da4d8c received from {username='slave'} at 127.0.186.8:50489
I20240503 14:13:35.533519 6820 tablet_copy_service.cc:433] P f1a669e28f014fe287932acf56bebc13: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-34773b2ab9194e70b3e49357a2da4d8c on tablet 34773b2ab9194e70b3e49357a2da4d8c with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:35.533838 9468 leader_election.cc:302] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:35.534057 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad2f2ef32d4fc88cdd62bfeab97b8a" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:35.534149 11806 raft_consensus.cc:2781] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.534232 11806 raft_consensus.cc:482] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.534296 9929 raft_consensus.cc:2454] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:35.534349 11806 raft_consensus.cc:3037] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.536787 11806 raft_consensus.cc:504] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.537346 11806 leader_election.cc:288] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:35.538030 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad2f2ef32d4fc88cdd62bfeab97b8a" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:35.538183 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10ad2f2ef32d4fc88cdd62bfeab97b8a" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:35.538275 9929 raft_consensus.cc:3037] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.538429 9798 raft_consensus.cc:3037] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.541272 9798 raft_consensus.cc:2454] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:35.541759 9468 leader_election.cc:302] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:35.542083 11806 raft_consensus.cc:2781] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.542236 11806 raft_consensus.cc:686] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:35.542742 9929 raft_consensus.cc:2454] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:35.542625 11806 consensus_queue.cc:237] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:35.546725 6655 catalog_manager.cc:5484] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.560534 11977 raft_consensus.cc:482] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:35.560706 11977 raft_consensus.cc:504] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.561349 11977 leader_election.cc:288] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:35.561911 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005dec40a9054f7fa95ea8722e754e24" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:35.561888 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005dec40a9054f7fa95ea8722e754e24" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:35.562132 7191 raft_consensus.cc:2454] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:35.562132 9531 raft_consensus.cc:2454] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:35.563045 9864 leader_election.cc:302] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:35.563333 11977 raft_consensus.cc:2781] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:35.563524 11977 raft_consensus.cc:482] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:35.563644 11977 raft_consensus.cc:3037] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.566078 11977 raft_consensus.cc:504] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.566615 11977 leader_election.cc:288] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:35.567178 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005dec40a9054f7fa95ea8722e754e24" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:35.567390 9531 raft_consensus.cc:3037] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.567364 7191 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "005dec40a9054f7fa95ea8722e754e24" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:35.567629 7191 raft_consensus.cc:3037] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:35.569527 9531 raft_consensus.cc:2454] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:35.569553 7191 raft_consensus.cc:2454] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:35.569924 9865 leader_election.cc:302] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:35.570199 11977 raft_consensus.cc:2781] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:35.570438 11977 raft_consensus.cc:686] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:35.570712 11977 consensus_queue.cc:237] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:35.574157 6655 catalog_manager.cc:5484] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.627753 6655 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:35.636063 7060 raft_consensus.cc:1261] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:35.636073 9531 raft_consensus.cc:1261] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:35.638415 11225 consensus_queue.cc:1035] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.639811 11225 consensus_queue.cc:1035] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.644210 12209 raft_consensus.cc:1050] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
I20240503 14:13:35.645102 12209 consensus_queue.cc:237] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:35.660151 7191 raft_consensus.cc:1261] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:35.660635 9530 raft_consensus.cc:1261] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:35.661217 11225 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:35.662503 11225 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:35.669420 7058 raft_consensus.cc:1261] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:35.672991 12209 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:35.674003 12209 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:35.688460 6654 catalog_manager.cc:5484] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:35.692363 9530 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:35.702370 7191 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:35.703267 6655 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet af479b9027c84a1680b30ccf44b464d1 (table rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c]). Expected version 1 got 0
I20240503 14:13:35.716782 9662 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:35.716889 6655 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet af479b9027c84a1680b30ccf44b464d1 (table rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c]). Expected version 1 got 0
I20240503 14:13:35.717625 11806 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.724664 9798 consensus_queue.cc:237] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:35.729113 7191 raft_consensus.cc:1261] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:35.729792 11225 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.730356 12226 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:35.730990 9530 raft_consensus.cc:1261] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:35.731163 12226 tablet.cc:1620] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:35.731524 11225 consensus_queue.cc:1035] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.732566 12209 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:35.735211 12231 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:35.736003 12231 tablet.cc:1620] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:35.737425 12232 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:35.737891 7188 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:35.738976 12232 tablet.cc:1620] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:35.746977 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7760fdb06dac4a46ab4452a67d7110aa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:35.736917 9533 raft_consensus.cc:2932] T 7760fdb06dac4a46ab4452a67d7110aa P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:35.747000 6655 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet af479b9027c84a1680b30ccf44b464d1 (table rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c]). Expected version 1 got 0
I20240503 14:13:35.751828 6653 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet af479b9027c84a1680b30ccf44b464d1 (table rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c]). Expected version 1 got 0
I20240503 14:13:35.753834 6654 catalog_manager.cc:5484] T 7760fdb06dac4a46ab4452a67d7110aa P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:35.756428 11816 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.759235 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 7760fdb06dac4a46ab4452a67d7110aa with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:35.759627 12193 tablet_replica.cc:330] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:35.759853 12193 raft_consensus.cc:2229] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:35.760023 12193 pending_rounds.cc:62] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: Trying to abort 1 pending ops.
I20240503 14:13:35.760116 12193 pending_rounds.cc:68] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023598041680048128 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "7760fdb06dac4a46ab4452a67d7110aa" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } } }
I20240503 14:13:35.760540 12193 raft_consensus.cc:2894] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 14:13:35.760679 12193 raft_consensus.cc:2866] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:35.760790 12193 raft_consensus.cc:2258] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:35.761615 12193 ts_tablet_manager.cc:1875] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:35.775091 12193 ts_tablet_manager.cc:1888] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:35.776589 12193 log.cc:1196] T 7760fdb06dac4a46ab4452a67d7110aa P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/7760fdb06dac4a46ab4452a67d7110aa
I20240503 14:13:35.777483 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 7760fdb06dac4a46ab4452a67d7110aa (table rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]) successfully deleted
I20240503 14:13:35.780254 12035 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:35.786198 6655 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c] alter complete (version 1)
I20240503 14:13:35.794579 9929 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:35.795295 11806 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.815623 7191 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:35.816393 11996 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:35.829303 12178 raft_consensus.cc:1050] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:35.830428 12178 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:35.845011 7060 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:35.845605 12178 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:35.846107 7191 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:35.847047 12186 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:35.853377 9662 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:35.855648 12178 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:35.864746 12186 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:35.868237 9929 raft_consensus.cc:1203] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:35.880424 7060 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:35.897533 6653 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:35.914476 6653 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:35.928493 12226 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:35.929092 12226 tablet.cc:1620] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:35.929793 12231 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:35.930238 12231 tablet.cc:1620] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:35.947069 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=7b809ce1c952461abb43a5e4ddbf2c6c] alter complete (version 2)
I20240503 14:13:35.951725 9662 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:35.961988 7191 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:35.977519 6929 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:35.984009 9663 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:35.989162 12186 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:35.994217 12186 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
W20240503 14:13:35.995415 7126 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 79159.4us
I20240503 14:13:35.995172 9663 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:35.997907 7188 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:36.000944 12232 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:36.002472 12232 tablet.cc:1620] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:36.003948 12186 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:36.005753 6653 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:36.006476 7191 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:36.006898 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b154f780fc742fa9b0efd609c307f95 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:36.015177 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b154f780fc742fa9b0efd609c307f95 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:36.015724 12193 tablet_replica.cc:330] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:36.016064 12193 raft_consensus.cc:2229] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.016381 12193 raft_consensus.cc:2258] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.018043 12193 ts_tablet_manager.cc:1875] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:36.025606 12193 ts_tablet_manager.cc:1888] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:36.025765 12193 log.cc:1196] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b154f780fc742fa9b0efd609c307f95
I20240503 14:13:36.026487 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 9b154f780fc742fa9b0efd609c307f95 (table rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]) successfully deleted
I20240503 14:13:36.056068 12015 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.057103 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:36.064585 11977 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.065179 12015 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.073429 12226 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.076385 12232 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.079718 12231 tablet.cc:1722] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:36.087272 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=7b809ce1c952461abb43a5e4ddbf2c6c] alter complete (version 3)
I20240503 14:13:36.098578 11976 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.112128 6653 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:36.124827 7040 tablet_service.cc:1467] Processing CreateTablet for tablet a9e7abf8b1cb44459dd0b103b786ff21 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.125005 9908 tablet_service.cc:1467] Processing CreateTablet for tablet a9e7abf8b1cb44459dd0b103b786ff21 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.125365 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e7abf8b1cb44459dd0b103b786ff21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.125399 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e7abf8b1cb44459dd0b103b786ff21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.128211 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 8811b896c0504775b03baf4925026adf (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.128732 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8811b896c0504775b03baf4925026adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.131312 12295 tablet_bootstrap.cc:492] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:36.132037 9779 tablet_service.cc:1467] Processing CreateTablet for tablet a9e7abf8b1cb44459dd0b103b786ff21 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.132475 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9e7abf8b1cb44459dd0b103b786ff21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.132416 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 8811b896c0504775b03baf4925026adf (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.132601 9909 tablet_service.cc:1467] Processing CreateTablet for tablet b97ddea52b624c668cb0b28ab0f38196 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.132875 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8811b896c0504775b03baf4925026adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.132995 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97ddea52b624c668cb0b28ab0f38196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.133044 12295 tablet_bootstrap.cc:654] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.135959 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 8811b896c0504775b03baf4925026adf (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.136423 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8811b896c0504775b03baf4925026adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.140759 6910 tablet_service.cc:1467] Processing CreateTablet for tablet b97ddea52b624c668cb0b28ab0f38196 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.141237 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97ddea52b624c668cb0b28ab0f38196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.149673 9644 tablet_service.cc:1467] Processing CreateTablet for tablet b97ddea52b624c668cb0b28ab0f38196 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.150211 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97ddea52b624c668cb0b28ab0f38196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.157603 12298 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:36.160555 12298 tablet_bootstrap.cc:654] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.163508 12302 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:36.170200 12302 tablet_bootstrap.cc:654] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.170869 12298 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:36.171049 12297 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:36.171723 12298 ts_tablet_manager.cc:1367] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240503 14:13:36.172995 12298 raft_consensus.cc:348] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.174719 12298 raft_consensus.cc:374] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.174986 12298 raft_consensus.cc:729] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.175446 12298 consensus_queue.cc:260] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.178776 12297 tablet_bootstrap.cc:654] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.181977 12298 ts_tablet_manager.cc:1398] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:13:36.182263 12302 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:36.182523 12302 ts_tablet_manager.cc:1367] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.010s
I20240503 14:13:36.185575 12299 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:36.186350 12295 tablet_bootstrap.cc:492] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:36.186578 12295 ts_tablet_manager.cc:1367] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.055s user 0.021s sys 0.013s
I20240503 14:13:36.187026 12302 raft_consensus.cc:348] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.187278 12302 raft_consensus.cc:374] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.187498 12302 raft_consensus.cc:729] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.187824 12302 consensus_queue.cc:260] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.188550 12302 ts_tablet_manager.cc:1398] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:36.188095 12295 raft_consensus.cc:348] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.189484 12295 raft_consensus.cc:374] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.189595 12295 raft_consensus.cc:729] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.189832 12295 consensus_queue.cc:260] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.190479 12295 ts_tablet_manager.cc:1398] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:36.196370 12297 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:36.196630 12297 ts_tablet_manager.cc:1367] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20240503 14:13:36.198230 12297 raft_consensus.cc:348] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.198474 12297 raft_consensus.cc:374] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.198650 12297 raft_consensus.cc:729] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.198758 6651 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:36.199025 6651 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:36.198911 12297 consensus_queue.cc:260] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.199685 12297 ts_tablet_manager.cc:1398] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:36.200088 12297 tablet_bootstrap.cc:492] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:36.201761 12297 tablet_bootstrap.cc:654] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.202517 6651 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet af479b9027c84a1680b30ccf44b464d1
I20240503 14:13:36.203441 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet af479b9027c84a1680b30ccf44b464d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:36.203775 12193 tablet_replica.cc:330] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:36.203944 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet af479b9027c84a1680b30ccf44b464d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:36.203972 12193 raft_consensus.cc:2229] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.204195 12193 raft_consensus.cc:2258] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.205327 12193 ts_tablet_manager.cc:1875] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.209594 12292 raft_consensus.cc:482] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.209808 12292 raft_consensus.cc:504] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.211012 12292 leader_election.cc:288] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:36.211301 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet af479b9027c84a1680b30ccf44b464d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:36 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:36.211441 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:36.211550 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:36.211691 9663 raft_consensus.cc:2454] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:36.211827 6929 raft_consensus.cc:2454] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:36.212110 9863 leader_election.cc:302] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:36.212648 12292 raft_consensus.cc:2781] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:36.212785 12292 raft_consensus.cc:482] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:36.212879 12292 raft_consensus.cc:3037] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.213078 12193 ts_tablet_manager.cc:1888] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.213177 12193 log.cc:1196] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/af479b9027c84a1680b30ccf44b464d1
I20240503 14:13:36.213490 12193 ts_tablet_manager.cc:1909] T af479b9027c84a1680b30ccf44b464d1 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:36.214407 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet af479b9027c84a1680b30ccf44b464d1 (table rebalancer_dynamic_table_new_name [id=7b809ce1c952461abb43a5e4ddbf2c6c]) successfully deleted
I20240503 14:13:36.216506 12292 raft_consensus.cc:504] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.217355 12292 leader_election.cc:288] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:36.217792 12301 tablet_bootstrap.cc:492] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:36.217885 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:36.218111 6929 raft_consensus.cc:3037] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.218170 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:36.218369 9663 raft_consensus.cc:3037] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.220938 9663 raft_consensus.cc:2454] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:36.220947 6929 raft_consensus.cc:2454] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:36.222992 12297 tablet_bootstrap.cc:492] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:36.223220 9863 leader_election.cc:302] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:36.223688 12299 tablet_bootstrap.cc:654] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.223750 12297 ts_tablet_manager.cc:1367] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.000s
I20240503 14:13:36.224133 12292 raft_consensus.cc:2781] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:36.224728 12292 raft_consensus.cc:686] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:36.224819 12297 raft_consensus.cc:348] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.225155 12297 raft_consensus.cc:374] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.225095 12292 consensus_queue.cc:237] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.225291 12297 raft_consensus.cc:729] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.225682 12297 consensus_queue.cc:260] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.226377 12297 ts_tablet_manager.cc:1398] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:36.226728 12297 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:36.228526 12297 tablet_bootstrap.cc:654] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.236152 12301 tablet_bootstrap.cc:654] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.237532 12306 tablet_replica.cc:330] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:36.239492 6651 catalog_manager.cc:5484] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:36.240381 12307 tablet_replica.cc:330] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:36.241307 12306 raft_consensus.cc:2229] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.242789 12306 raft_consensus.cc:2258] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.243057 12307 raft_consensus.cc:2229] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:36.243492 12307 raft_consensus.cc:2258] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.244227 12299 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:36.244390 12299 ts_tablet_manager.cc:1367] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.003s
I20240503 14:13:36.244824 12306 ts_tablet_manager.cc:1875] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.245205 12299 raft_consensus.cc:348] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.245438 12299 raft_consensus.cc:374] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.245533 12299 raft_consensus.cc:729] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.245757 12299 consensus_queue.cc:260] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.246795 12297 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:36.246953 12297 ts_tablet_manager.cc:1367] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.006s
I20240503 14:13:36.247774 12297 raft_consensus.cc:348] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.247989 12297 raft_consensus.cc:374] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.248087 12297 raft_consensus.cc:729] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.248342 12297 consensus_queue.cc:260] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.248953 12299 ts_tablet_manager.cc:1398] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 14:13:36.249433 12292 raft_consensus.cc:482] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.249603 12292 raft_consensus.cc:504] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.250577 12292 leader_election.cc:288] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:36.250967 12297 ts_tablet_manager.cc:1398] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 14:13:36.250993 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8811b896c0504775b03baf4925026adf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:36.251211 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8811b896c0504775b03baf4925026adf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:36.251332 6800 raft_consensus.cc:2454] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
W20240503 14:13:36.251766 9864 leader_election.cc:341] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:36.251948 9863 leader_election.cc:302] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters: 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:36.252368 12292 raft_consensus.cc:2781] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:36.252465 12292 raft_consensus.cc:482] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:36.252568 12292 raft_consensus.cc:3037] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.255304 12292 raft_consensus.cc:504] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.256361 12292 leader_election.cc:288] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:36.257289 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8811b896c0504775b03baf4925026adf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:36.257557 6800 raft_consensus.cc:3037] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.257970 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8811b896c0504775b03baf4925026adf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
W20240503 14:13:36.258498 9864 leader_election.cc:341] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:36.260231 6800 raft_consensus.cc:2454] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:36.260736 9863 leader_election.cc:302] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters: 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:36.261197 12292 raft_consensus.cc:2781] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:36.261415 12292 raft_consensus.cc:686] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:36.261758 12292 consensus_queue.cc:237] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.262194 12307 ts_tablet_manager.cc:1875] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:36.266009 12306 ts_tablet_manager.cc:1888] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.266161 12306 log.cc:1196] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/af479b9027c84a1680b30ccf44b464d1
I20240503 14:13:36.266623 12306 ts_tablet_manager.cc:1909] T af479b9027c84a1680b30ccf44b464d1 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:36.267822 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet af479b9027c84a1680b30ccf44b464d1 (table rebalancer_dynamic_table_new_name [id=7b809ce1c952461abb43a5e4ddbf2c6c]) successfully deleted
I20240503 14:13:36.271481 11977 raft_consensus.cc:482] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.271683 11977 raft_consensus.cc:504] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.272578 11977 leader_election.cc:288] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:36.273217 12301 tablet_bootstrap.cc:492] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:36.273547 12301 ts_tablet_manager.cc:1367] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.074s user 0.000s sys 0.010s
I20240503 14:13:36.273650 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e7abf8b1cb44459dd0b103b786ff21" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:36.274516 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e7abf8b1cb44459dd0b103b786ff21" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:36.274756 7060 raft_consensus.cc:2454] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:36.275207 9865 leader_election.cc:302] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:36.275377 12301 raft_consensus.cc:348] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.275884 12301 raft_consensus.cc:374] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 14:13:36.275944 9864 leader_election.cc:341] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:36.276027 12301 raft_consensus.cc:729] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.276414 12301 consensus_queue.cc:260] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.277698 11977 raft_consensus.cc:2781] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:36.277817 11977 raft_consensus.cc:482] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:36.277901 11977 raft_consensus.cc:3037] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.280999 12301 ts_tablet_manager.cc:1398] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 14:13:36.281615 11977 raft_consensus.cc:504] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.283324 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e7abf8b1cb44459dd0b103b786ff21" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:36.283521 9798 raft_consensus.cc:3037] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.286379 9798 raft_consensus.cc:2454] T a9e7abf8b1cb44459dd0b103b786ff21 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:36.286803 12301 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:36.288234 9864 leader_election.cc:302] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:36.288791 11976 raft_consensus.cc:2781] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:36.289860 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9e7abf8b1cb44459dd0b103b786ff21" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:36.290092 7060 raft_consensus.cc:3037] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.292940 7060 raft_consensus.cc:2454] T a9e7abf8b1cb44459dd0b103b786ff21 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:36.294627 12307 ts_tablet_manager.cc:1888] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:36.295032 12307 log.cc:1196] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/af479b9027c84a1680b30ccf44b464d1
I20240503 14:13:36.294898 12301 tablet_bootstrap.cc:654] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.295573 12307 ts_tablet_manager.cc:1909] T af479b9027c84a1680b30ccf44b464d1 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:36.300047 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet af479b9027c84a1680b30ccf44b464d1 (table rebalancer_dynamic_table_new_name [id=7b809ce1c952461abb43a5e4ddbf2c6c]) successfully deleted
I20240503 14:13:36.304314 11977 leader_election.cc:288] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:36.305542 12301 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:36.305819 12301 ts_tablet_manager.cc:1367] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.003s
I20240503 14:13:36.307663 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:36.316174 11976 raft_consensus.cc:686] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:36.316709 11976 consensus_queue.cc:237] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.317953 6654 catalog_manager.cc:5484] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:36.319147 6780 tablet_service.cc:1467] Processing CreateTablet for tablet bf4d0161a6d84bb0a53188c45df762fd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266]), partition=
I20240503 14:13:36.319332 7040 tablet_service.cc:1467] Processing CreateTablet for tablet bf4d0161a6d84bb0a53188c45df762fd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266]), partition=
I20240503 14:13:36.319622 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4d0161a6d84bb0a53188c45df762fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.319520 9779 tablet_service.cc:1467] Processing CreateTablet for tablet bf4d0161a6d84bb0a53188c45df762fd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266]), partition=
I20240503 14:13:36.319725 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4d0161a6d84bb0a53188c45df762fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.319943 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4d0161a6d84bb0a53188c45df762fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.328648 12299 tablet_bootstrap.cc:492] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:36.330258 12299 tablet_bootstrap.cc:654] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.331434 12301 raft_consensus.cc:348] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.331794 12301 raft_consensus.cc:374] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.332288 12301 raft_consensus.cc:729] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.332587 12301 consensus_queue.cc:260] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.334640 12295 tablet_bootstrap.cc:492] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:36.335637 12299 tablet_bootstrap.cc:492] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:36.335834 12299 ts_tablet_manager.cc:1367] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240503 14:13:36.336222 12295 tablet_bootstrap.cc:654] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.336907 12299 raft_consensus.cc:348] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.337208 12299 raft_consensus.cc:374] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.337291 12299 raft_consensus.cc:729] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.337549 12299 consensus_queue.cc:260] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.338099 12299 ts_tablet_manager.cc:1398] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:36.338308 12301 ts_tablet_manager.cc:1398] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240503 14:13:36.344647 6653 catalog_manager.cc:5484] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:36.353014 12301 tablet_bootstrap.cc:492] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:36.355811 12301 tablet_bootstrap.cc:654] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.357236 9533 consensus_queue.cc:237] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:36.360389 7191 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:36.361303 12015 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:36.363258 9663 raft_consensus.cc:1261] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:36.367646 12015 consensus_queue.cc:1035] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:36.370790 12015 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:36.376561 9663 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:36.387261 7188 raft_consensus.cc:2932] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:36.396436 12252 raft_consensus.cc:1050] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:36.397020 6651 catalog_manager.cc:5484] T 32d91b6f4bb244c78bc6fef8b837f964 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:36.397603 12252 consensus_queue.cc:237] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:36.402331 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32d91b6f4bb244c78bc6fef8b837f964 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:36.402874 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 32d91b6f4bb244c78bc6fef8b837f964 with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:36.403779 12324 tablet_replica.cc:330] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:36.404109 12324 raft_consensus.cc:2229] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.406324 12295 tablet_bootstrap.cc:492] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:36.406635 12295 ts_tablet_manager.cc:1367] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.072s user 0.007s sys 0.000s
I20240503 14:13:36.407747 12295 raft_consensus.cc:348] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.408059 12295 raft_consensus.cc:374] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.408152 12295 raft_consensus.cc:729] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.408455 12295 consensus_queue.cc:260] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.409150 12295 ts_tablet_manager.cc:1398] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:36.418179 12301 tablet_bootstrap.cc:492] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:36.418344 12301 ts_tablet_manager.cc:1367] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.065s user 0.021s sys 0.019s
I20240503 14:13:36.418999 12301 raft_consensus.cc:348] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.419179 12301 raft_consensus.cc:374] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.419288 12301 raft_consensus.cc:729] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.419502 12301 consensus_queue.cc:260] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.419965 12324 raft_consensus.cc:2258] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.420145 12301 ts_tablet_manager.cc:1398] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:36.423120 12324 ts_tablet_manager.cc:1875] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:36.430371 6929 raft_consensus.cc:1261] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:36.431000 9929 raft_consensus.cc:1261] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:36.431051 12066 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:36.431648 12066 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:36.433053 12324 ts_tablet_manager.cc:1888] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 14:13:36.433188 12324 log.cc:1196] T 32d91b6f4bb244c78bc6fef8b837f964 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/32d91b6f4bb244c78bc6fef8b837f964
I20240503 14:13:36.434229 7058 raft_consensus.cc:1261] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:36.435096 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 32d91b6f4bb244c78bc6fef8b837f964 (table rebalancer_extra_table_2 [id=a5259ab5511342b0981bffe2e1fd8e62]) successfully deleted
I20240503 14:13:36.435763 12252 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:36.452709 12327 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.453871 9929 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.458276 6929 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.461831 6651 catalog_manager.cc:5484] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.463761 7058 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.470701 12252 raft_consensus.cc:482] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.470913 12252 raft_consensus.cc:504] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.471468 12252 leader_election.cc:288] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:36.471731 6800 consensus_queue.cc:237] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:36.471969 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4d0161a6d84bb0a53188c45df762fd" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:36.472198 9798 raft_consensus.cc:2454] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:36.473109 6735 leader_election.cc:302] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:36.474478 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4d0161a6d84bb0a53188c45df762fd" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:36.474705 7060 raft_consensus.cc:2454] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:36.475394 12327 raft_consensus.cc:2781] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:36.475559 12327 raft_consensus.cc:482] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:36.475719 12327 raft_consensus.cc:3037] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.476364 9929 raft_consensus.cc:1261] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:36.477064 6930 raft_consensus.cc:1261] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:36.477432 12252 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:36.477984 12252 consensus_queue.cc:1035] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:36.479660 12327 raft_consensus.cc:504] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.480904 12327 leader_election.cc:288] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:36.481463 6930 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.480652 12252 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.483453 9929 raft_consensus.cc:2932] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.491263 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4d0161a6d84bb0a53188c45df762fd" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:36.491503 7058 raft_consensus.cc:3037] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.492017 6654 catalog_manager.cc:5484] T 34773b2ab9194e70b3e49357a2da4d8c P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 5 to 6, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:36.494989 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 34773b2ab9194e70b3e49357a2da4d8c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:36.497709 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 34773b2ab9194e70b3e49357a2da4d8c with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:36.498039 12193 tablet_replica.cc:330] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:36.498265 12193 raft_consensus.cc:2229] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:36.498617 12193 raft_consensus.cc:2258] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:36.500118 12193 ts_tablet_manager.cc:1875] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:36.500361 7058 raft_consensus.cc:2454] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:36.501001 6736 leader_election.cc:302] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:36.501348 12327 raft_consensus.cc:2781] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:36.501525 12327 raft_consensus.cc:686] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:36.501838 12327 consensus_queue.cc:237] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.507057 12193 ts_tablet_manager.cc:1888] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 14:13:36.507231 12193 log.cc:1196] T 34773b2ab9194e70b3e49357a2da4d8c P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/34773b2ab9194e70b3e49357a2da4d8c
I20240503 14:13:36.508226 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 34773b2ab9194e70b3e49357a2da4d8c (table rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]) successfully deleted
I20240503 14:13:36.511082 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4d0161a6d84bb0a53188c45df762fd" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:36.511350 9798 raft_consensus.cc:3037] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.512073 6651 catalog_manager.cc:5484] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:36.514021 9798 raft_consensus.cc:2454] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:36.520840 11225 raft_consensus.cc:482] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:36.521068 11225 raft_consensus.cc:504] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:36.521816 11225 leader_election.cc:288] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:36.522819 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8811b896c0504775b03baf4925026adf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:36.523468 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8811b896c0504775b03baf4925026adf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:36.523775 6800 raft_consensus.cc:2379] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3efdbd3506274dba846ae40404f0572c in current term 1: Already voted for candidate ea801601713a4d55aee0de9e3473e0bc in this term.
I20240503 14:13:36.524307 9733 leader_election.cc:302] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:36.524603 11225 raft_consensus.cc:3037] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:36.527128 11225 raft_consensus.cc:2726] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:36.619290 12313 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.629729 11977 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.641716 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:36.669811 7058 raft_consensus.cc:1261] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:36.670851 12066 consensus_queue.cc:1035] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.672606 9798 raft_consensus.cc:1261] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:36.673589 12066 consensus_queue.cc:1035] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.686259 12313 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.721354 11976 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.759486 6654 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet bf4d0161a6d84bb0a53188c45df762fd (table rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266]). Expected version 1 got 0
I20240503 14:13:36.768297 11976 consensus_queue.cc:1035] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.773214 6653 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet bf4d0161a6d84bb0a53188c45df762fd (table rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266]). Expected version 1 got 0
I20240503 14:13:36.790498 12388 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:36.794344 12388 tablet.cc:1620] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:36.795902 6653 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet bf4d0161a6d84bb0a53188c45df762fd (table rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266]). Expected version 1 got 0
I20240503 14:13:36.803081 11977 consensus_queue.cc:1035] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:36.808136 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266] alter complete (version 1)
I20240503 14:13:36.822729 12385 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:36.823402 12386 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:36.825505 12386 tablet.cc:1620] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:36.828900 6654 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet bf4d0161a6d84bb0a53188c45df762fd (table rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266]). Expected version 1 got 0
I20240503 14:13:36.829623 6651 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet bf4d0161a6d84bb0a53188c45df762fd (table rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266]). Expected version 1 got 0
I20240503 14:13:36.826304 12385 tablet.cc:1620] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:36.868602 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:36.878862 6780 tablet_service.cc:1467] Processing CreateTablet for tablet d1df9d5b8a3d4a489cbccca67b223168 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.880250 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5bfc2d9491467caa7854c4b4c3ddf6 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.880693 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5bfc2d9491467caa7854c4b4c3ddf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.881222 9909 tablet_service.cc:1467] Processing CreateTablet for tablet d1df9d5b8a3d4a489cbccca67b223168 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.881719 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1df9d5b8a3d4a489cbccca67b223168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.882032 7040 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed8f0fd81a43f288295c58c22ea08c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.882443 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed8f0fd81a43f288295c58c22ea08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.884264 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5bfc2d9491467caa7854c4b4c3ddf6 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.884567 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 7c5bfc2d9491467caa7854c4b4c3ddf6 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.884621 9511 tablet_service.cc:1467] Processing CreateTablet for tablet d1df9d5b8a3d4a489cbccca67b223168 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.884817 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5bfc2d9491467caa7854c4b4c3ddf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.885013 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c5bfc2d9491467caa7854c4b4c3ddf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.887758 9910 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed8f0fd81a43f288295c58c22ea08c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.888191 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1df9d5b8a3d4a489cbccca67b223168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.888252 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed8f0fd81a43f288295c58c22ea08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.889182 9779 tablet_service.cc:1467] Processing CreateTablet for tablet a2ed8f0fd81a43f288295c58c22ea08c (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:36.889676 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2ed8f0fd81a43f288295c58c22ea08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.894443 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1df9d5b8a3d4a489cbccca67b223168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:36.901054 12295 tablet_bootstrap.cc:492] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:36.902776 12295 tablet_bootstrap.cc:654] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.930166 12398 tablet_bootstrap.cc:492] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:36.933077 12404 tablet_bootstrap.cc:492] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:36.933862 12401 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:36.936424 12404 tablet_bootstrap.cc:654] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.936723 12401 tablet_bootstrap.cc:654] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.937083 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:36.938689 12295 tablet_bootstrap.cc:492] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:36.939031 12295 ts_tablet_manager.cc:1367] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.008s
I20240503 14:13:36.939437 12397 tablet_bootstrap.cc:492] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:36.940284 12295 raft_consensus.cc:348] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.940644 12295 raft_consensus.cc:374] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.940773 12295 raft_consensus.cc:729] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.941087 12295 consensus_queue.cc:260] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:36.941272 12398 tablet_bootstrap.cc:654] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.945858 12405 tablet_bootstrap.cc:492] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:36.952020 12397 tablet_bootstrap.cc:654] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.955027 12399 tablet_bootstrap.cc:492] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:36.957144 12388 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:36.957433 12399 tablet_bootstrap.cc:654] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.957679 12388 tablet.cc:1620] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:36.959647 12398 tablet_bootstrap.cc:492] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:36.959872 12398 ts_tablet_manager.cc:1367] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 14:13:36.960595 12398 raft_consensus.cc:348] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.963343 12399 tablet_bootstrap.cc:492] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:36.963985 12398 raft_consensus.cc:374] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.964085 12398 raft_consensus.cc:729] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.964309 12399 ts_tablet_manager.cc:1367] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 14:13:36.964329 12398 consensus_queue.cc:260] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.966962 12398 ts_tablet_manager.cc:1398] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:36.968227 12386 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:36.968715 12386 tablet.cc:1620] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:36.968760 12404 tablet_bootstrap.cc:492] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:36.968904 12404 ts_tablet_manager.cc:1367] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240503 14:13:36.969731 12404 raft_consensus.cc:348] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.969941 12404 raft_consensus.cc:374] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:36.970021 12404 raft_consensus.cc:729] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:36.970018 12385 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:36.970221 12404 consensus_queue.cc:260] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:36.970546 12385 tablet.cc:1620] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:36.970834 12404 ts_tablet_manager.cc:1398] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:36.971190 12404 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:36.978986 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=e2af0c19d8c94f248d51100737cd9266] alter complete (version 2)
I20240503 14:13:36.987901 12405 tablet_bootstrap.cc:654] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.988058 12404 tablet_bootstrap.cc:654] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:36.988516 12295 ts_tablet_manager.cc:1398] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20240503 14:13:37.002261 12404 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:37.002535 12404 ts_tablet_manager.cc:1367] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.004s
I20240503 14:13:37.004024 12404 raft_consensus.cc:348] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.004381 12404 raft_consensus.cc:374] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.004510 12404 raft_consensus.cc:729] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.006435 12401 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:37.009510 12404 consensus_queue.cc:260] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.015753 12404 ts_tablet_manager.cc:1398] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:37.017840 12399 raft_consensus.cc:348] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.016148 12401 ts_tablet_manager.cc:1367] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20240503 14:13:37.021416 12401 raft_consensus.cc:348] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.022958 12401 raft_consensus.cc:374] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.023264 12401 raft_consensus.cc:729] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.023941 12401 consensus_queue.cc:260] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.026188 12399 raft_consensus.cc:374] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.026346 12399 raft_consensus.cc:729] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.026726 12399 consensus_queue.cc:260] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.027276 12405 tablet_bootstrap.cc:492] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:37.027465 12399 ts_tablet_manager.cc:1398] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240503 14:13:37.027505 12405 ts_tablet_manager.cc:1367] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.003s
I20240503 14:13:37.028718 12405 raft_consensus.cc:348] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.029439 12401 ts_tablet_manager.cc:1398] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 14:13:37.030222 12405 raft_consensus.cc:374] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.033654 12397 tablet_bootstrap.cc:492] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:37.033691 12399 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:37.033862 12397 ts_tablet_manager.cc:1367] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.095s user 0.011s sys 0.000s
I20240503 14:13:37.034724 12397 raft_consensus.cc:348] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:37.034982 12397 raft_consensus.cc:374] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.035149 12397 raft_consensus.cc:729] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.035403 12397 consensus_queue.cc:260] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:37.036124 12397 ts_tablet_manager.cc:1398] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:37.037139 12399 tablet_bootstrap.cc:654] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.030516 12405 raft_consensus.cc:729] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.043337 12405 consensus_queue.cc:260] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.043973 12313 raft_consensus.cc:482] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.044143 12313 raft_consensus.cc:504] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.044997 12313 leader_election.cc:288] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:37.045933 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed8f0fd81a43f288295c58c22ea08c" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:37.046166 7060 raft_consensus.cc:2454] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:37.046769 9795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed8f0fd81a43f288295c58c22ea08c" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:37.051223 9795 raft_consensus.cc:2454] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:37.053898 12405 ts_tablet_manager.cc:1398] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20240503 14:13:37.060819 9865 leader_election.cc:302] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:37.064675 11977 raft_consensus.cc:2781] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.064836 11977 raft_consensus.cc:482] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.064944 11977 raft_consensus.cc:3037] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.076292 11977 raft_consensus.cc:504] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.086130 9795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed8f0fd81a43f288295c58c22ea08c" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:37.086436 9795 raft_consensus.cc:3037] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.090205 6654 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:37.091260 9795 raft_consensus.cc:2454] T a2ed8f0fd81a43f288295c58c22ea08c P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:37.113399 12385 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:37.125713 12386 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:37.139448 6654 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=e2af0c19d8c94f248d51100737cd9266] alter complete (version 3)
I20240503 14:13:37.119849 12388 tablet.cc:1722] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:37.141582 7058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2ed8f0fd81a43f288295c58c22ea08c" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:37.141834 7058 raft_consensus.cc:3037] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.144922 7058 raft_consensus.cc:2454] T a2ed8f0fd81a43f288295c58c22ea08c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:37.149436 9865 leader_election.cc:302] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:37.150236 12313 raft_consensus.cc:2781] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.150429 12313 raft_consensus.cc:686] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:37.150805 12313 consensus_queue.cc:237] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.159400 12341 raft_consensus.cc:482] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.159658 12341 raft_consensus.cc:504] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.160658 12341 leader_election.cc:288] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:37.162765 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df9d5b8a3d4a489cbccca67b223168" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
W20240503 14:13:37.167373 6734 leader_election.cc:341] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:37.171147 7040 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:13:37.174430 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df9d5b8a3d4a489cbccca67b223168" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:37.174755 9533 raft_consensus.cc:2454] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:37.177464 6736 leader_election.cc:302] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters: ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:37.178161 12327 raft_consensus.cc:2781] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.178305 12327 raft_consensus.cc:482] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.178409 12327 raft_consensus.cc:3037] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.181764 12327 raft_consensus.cc:504] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.223172 11977 leader_election.cc:288] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:37.239425 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df9d5b8a3d4a489cbccca67b223168" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
W20240503 14:13:37.240357 6734 leader_election.cc:341] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:37.241904 12327 leader_election.cc:288] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:37.244024 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df9d5b8a3d4a489cbccca67b223168" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:37.244252 9533 raft_consensus.cc:3037] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.247357 9533 raft_consensus.cc:2454] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:37.248502 6736 leader_election.cc:302] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters: ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:37.249073 12327 raft_consensus.cc:2781] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.250056 12327 raft_consensus.cc:686] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:37.250540 12327 consensus_queue.cc:237] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.254562 6910 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:13:37.262200 12399 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:37.262514 12399 ts_tablet_manager.cc:1367] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.229s user 0.046s sys 0.120s
I20240503 14:13:37.263573 12399 raft_consensus.cc:348] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.263958 12399 raft_consensus.cc:374] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.264092 12399 raft_consensus.cc:729] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.264899 12399 consensus_queue.cc:260] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.266099 12399 ts_tablet_manager.cc:1398] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:37.269939 6654 catalog_manager.cc:5484] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:37.292713 6653 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:37.292954 6653 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:37.297628 6653 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet bf4d0161a6d84bb0a53188c45df762fd
I20240503 14:13:37.299188 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet bf4d0161a6d84bb0a53188c45df762fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:37 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:37.299335 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet bf4d0161a6d84bb0a53188c45df762fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:37 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:37.299331 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet bf4d0161a6d84bb0a53188c45df762fd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:37 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:37.327962 12443 tablet_replica.cc:330] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:37.328485 12443 raft_consensus.cc:2229] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:37.328845 12443 raft_consensus.cc:2258] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:37.330324 12443 ts_tablet_manager.cc:1875] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:37.331745 7170 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 14:13:37.339236 9644 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240503 14:13:37.352452 12444 tablet_replica.cc:330] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:37.353413 12444 raft_consensus.cc:2229] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:37.354060 12444 raft_consensus.cc:2258] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:37.362818 12444 ts_tablet_manager.cc:1875] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:37.367990 6654 catalog_manager.cc:5484] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:37.371958 12313 raft_consensus.cc:482] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.372236 12313 raft_consensus.cc:504] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:37.373600 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df9d5b8a3d4a489cbccca67b223168" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:37.383018 6779 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:13:37.386020 12445 tablet_replica.cc:330] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:37.386246 12445 raft_consensus.cc:2229] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:37.386716 12445 raft_consensus.cc:2258] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:37.393800 12313 leader_election.cc:288] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:37.394256 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1df9d5b8a3d4a489cbccca67b223168" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:37.394585 9533 raft_consensus.cc:2379] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea801601713a4d55aee0de9e3473e0bc in current term 1: Already voted for candidate f1a669e28f014fe287932acf56bebc13 in this term.
I20240503 14:13:37.395095 9865 leader_election.cc:302] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:37.395535 12314 raft_consensus.cc:3037] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.396248 12445 ts_tablet_manager.cc:1875] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:37.400348 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:37.401945 12314 raft_consensus.cc:2726] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:37.402242 12411 raft_consensus.cc:482] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.402412 12411 raft_consensus.cc:504] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:37.403805 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5bfc2d9491467caa7854c4b4c3ddf6" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:37.404062 9533 raft_consensus.cc:2454] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:37.404675 7127 leader_election.cc:302] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:37.404960 12412 raft_consensus.cc:2781] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.405028 12412 raft_consensus.cc:482] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.405084 12412 raft_consensus.cc:3037] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.409806 12411 leader_election.cc:288] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:37.411047 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5bfc2d9491467caa7854c4b4c3ddf6" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:37.411345 9663 raft_consensus.cc:2454] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:37.413127 12412 raft_consensus.cc:504] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:37.416515 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5bfc2d9491467caa7854c4b4c3ddf6" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:37.416800 9533 raft_consensus.cc:3037] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.420337 12412 leader_election.cc:288] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:37.422237 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c5bfc2d9491467caa7854c4b4c3ddf6" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:37.422487 9663 raft_consensus.cc:3037] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.427294 12445 ts_tablet_manager.cc:1888] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:37.429263 12445 log.cc:1196] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/bf4d0161a6d84bb0a53188c45df762fd
I20240503 14:13:37.430101 12445 ts_tablet_manager.cc:1909] T bf4d0161a6d84bb0a53188c45df762fd P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:37.430311 9663 raft_consensus.cc:2454] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:37.432080 7125 leader_election.cc:302] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:37.432574 12263 raft_consensus.cc:2781] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.432715 12263 raft_consensus.cc:686] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:37.432960 12263 consensus_queue.cc:237] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:37.433529 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet bf4d0161a6d84bb0a53188c45df762fd (table rebalancer_dynamic_table_new_name [id=e2af0c19d8c94f248d51100737cd9266]) successfully deleted
I20240503 14:13:37.434621 9533 raft_consensus.cc:2454] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:37.436923 12443 ts_tablet_manager.cc:1888] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:37.442909 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7db2cf1f6a4d62bf5562125591fb51 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80]), partition=
I20240503 14:13:37.442914 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7db2cf1f6a4d62bf5562125591fb51 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80]), partition=
I20240503 14:13:37.443308 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7db2cf1f6a4d62bf5562125591fb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.443318 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7db2cf1f6a4d62bf5562125591fb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.442952 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7db2cf1f6a4d62bf5562125591fb51 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80]), partition=
I20240503 14:13:37.447440 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7db2cf1f6a4d62bf5562125591fb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:37.454607 12443 log.cc:1196] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/bf4d0161a6d84bb0a53188c45df762fd
I20240503 14:13:37.459861 12399 tablet_bootstrap.cc:492] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:37.461993 12399 tablet_bootstrap.cc:654] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.476975 12443 ts_tablet_manager.cc:1909] T bf4d0161a6d84bb0a53188c45df762fd P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:37.478305 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet bf4d0161a6d84bb0a53188c45df762fd (table rebalancer_dynamic_table_new_name [id=e2af0c19d8c94f248d51100737cd9266]) successfully deleted
I20240503 14:13:37.483258 12295 tablet_bootstrap.cc:492] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:37.484844 12295 tablet_bootstrap.cc:654] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:37.489689 9864 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 94431.1us
I20240503 14:13:37.490628 6653 catalog_manager.cc:5484] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:37.494642 12399 tablet_bootstrap.cc:492] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:37.495069 12399 ts_tablet_manager.cc:1367] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.002s
I20240503 14:13:37.496757 12399 raft_consensus.cc:348] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.497072 12399 raft_consensus.cc:374] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.498270 12399 raft_consensus.cc:729] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.498579 12399 consensus_queue.cc:260] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.499300 12399 ts_tablet_manager.cc:1398] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:37.508327 12449 tablet_bootstrap.cc:492] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:37.518469 12449 tablet_bootstrap.cc:654] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:37.524727 12295 tablet_bootstrap.cc:492] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:37.526460 12295 ts_tablet_manager.cc:1367] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240503 14:13:37.533036 9512 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 14:13:37.534951 12295 raft_consensus.cc:348] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.546813 12444 ts_tablet_manager.cc:1888] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:37.547116 12444 log.cc:1196] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/bf4d0161a6d84bb0a53188c45df762fd
I20240503 14:13:37.547855 12444 ts_tablet_manager.cc:1909] T bf4d0161a6d84bb0a53188c45df762fd P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:37.549444 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet bf4d0161a6d84bb0a53188c45df762fd (table rebalancer_dynamic_table_new_name [id=e2af0c19d8c94f248d51100737cd9266]) successfully deleted
W20240503 14:13:37.550546 9733 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 49338.5us
I20240503 14:13:37.535527 12295 raft_consensus.cc:374] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.555258 12295 raft_consensus.cc:729] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.555706 12295 consensus_queue.cc:260] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.558099 12295 ts_tablet_manager.cc:1398] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
W20240503 14:13:37.566140 9601 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.5:36609, user_credentials={real_user=slave}} blocked reactor thread for 53828.6us
I20240503 14:13:37.577503 9910 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240503 14:13:37.590140 12449 tablet_bootstrap.cc:492] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:37.590443 12449 ts_tablet_manager.cc:1367] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.002s
I20240503 14:13:37.591395 12449 raft_consensus.cc:348] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.591799 12449 raft_consensus.cc:374] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:37.591919 12449 raft_consensus.cc:729] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:37.592180 9779 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:13:37.592223 12449 consensus_queue.cc:260] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.593044 12449 ts_tablet_manager.cc:1398] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:37.596854 11977 consensus_queue.cc:1035] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.682873 12327 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.683574 11977 consensus_queue.cc:1035] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.715042 12341 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.728960 12292 raft_consensus.cc:482] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:37.729231 12292 raft_consensus.cc:504] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.730276 12292 leader_election.cc:288] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:37.731319 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7db2cf1f6a4d62bf5562125591fb51" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:37.731696 6930 raft_consensus.cc:2454] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:37.732507 9863 leader_election.cc:302] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:37.733739 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7db2cf1f6a4d62bf5562125591fb51" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:37.734051 7060 raft_consensus.cc:2454] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:37.734987 12292 raft_consensus.cc:2781] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:37.735136 12292 raft_consensus.cc:482] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:37.735245 12292 raft_consensus.cc:3037] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.741845 12292 raft_consensus.cc:504] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.743077 12292 leader_election.cc:288] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:37.743264 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7db2cf1f6a4d62bf5562125591fb51" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:37.743580 6930 raft_consensus.cc:3037] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.757133 6930 raft_consensus.cc:2454] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:37.758143 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e7db2cf1f6a4d62bf5562125591fb51" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:37.758414 7060 raft_consensus.cc:3037] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:37.758495 9863 leader_election.cc:302] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:37.759125 11977 raft_consensus.cc:2781] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:37.759291 11977 raft_consensus.cc:686] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:37.759572 11977 consensus_queue.cc:237] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:37.762037 7060 raft_consensus.cc:2454] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:37.785135 6654 catalog_manager.cc:5484] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:37.874754 9928 consensus_queue.cc:237] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:37.877091 6798 raft_consensus.cc:1261] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.877705 9664 raft_consensus.cc:1261] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.877661 12292 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:37.878345 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6f8dfcb870a4461b03415db9c33452e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.878453 12292 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:37.880676 12314 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:37.881203 6798 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:37.886531 6653 catalog_manager.cc:5484] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:37.891213 9664 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:37.900172 12263 consensus_queue.cc:1035] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.905599 6653 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:37.913203 7060 raft_consensus.cc:1261] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.913606 12412 consensus_queue.cc:1035] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.914136 12314 consensus_queue.cc:1035] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.914731 6930 raft_consensus.cc:1261] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.915444 12314 consensus_queue.cc:1035] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:37.917400 9797 consensus_queue.cc:237] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:37.927775 9928 raft_consensus.cc:1261] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:37.928562 9733 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89ff0991a4e245609405ff945388417b. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.928726 7058 raft_consensus.cc:1261] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.929257 12453 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:37.929636 11225 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:37.933894 6654 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 4e7db2cf1f6a4d62bf5562125591fb51 (table rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80]). Expected version 1 got 0
I20240503 14:13:37.936220 12453 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:37.942266 9928 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:37.944025 6653 catalog_manager.cc:5484] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:37.951283 7058 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:37.953058 12494 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:37.955070 6654 catalog_manager.cc:5576] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285) does not have the latest schema for tablet 4e7db2cf1f6a4d62bf5562125591fb51 (table rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80]). Expected version 1 got 0
I20240503 14:13:37.958149 12494 tablet.cc:1620] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:37.963338 12495 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:37.964000 12496 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:37.965118 12496 tablet.cc:1620] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: MemRowSet was empty: no flush needed.
I20240503 14:13:37.965219 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80] alter complete (version 1)
I20240503 14:13:37.967188 6653 catalog_manager.cc:5576] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285) does not have the latest schema for tablet 4e7db2cf1f6a4d62bf5562125591fb51 (table rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80]). Expected version 1 got 0
I20240503 14:13:37.971024 6654 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 4e7db2cf1f6a4d62bf5562125591fb51 (table rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80]). Expected version 1 got 0
I20240503 14:13:37.972780 12495 tablet.cc:1620] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:37.990888 6929 consensus_queue.cc:237] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
W20240503 14:13:37.994628 6866 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6d600a8b0134a018f8c9e5a1d9b79c5. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:37.994653 9662 raft_consensus.cc:1261] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.995198 12452 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:37.995393 9531 raft_consensus.cc:1261] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:37.996069 12452 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:37.998361 12452 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:38.000408 9662 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:38.001881 9533 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:38.007911 6654 catalog_manager.cc:5484] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:38.025642 6654 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:38.027704 6929 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:38.030434 7188 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:38.030532 9662 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:38.031085 12506 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240503 14:13:38.034988 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 27d8cb8f47304625999bb9a188001629 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.035449 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d8cb8f47304625999bb9a188001629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.036033 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 27d8cb8f47304625999bb9a188001629 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.036193 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 27d8cb8f47304625999bb9a188001629 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.035534 9644 tablet_service.cc:1467] Processing CreateTablet for tablet f2f54c0c48b848ed898534efef55fb62 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.036465 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d8cb8f47304625999bb9a188001629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.036638 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d8cb8f47304625999bb9a188001629. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.036695 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f54c0c48b848ed898534efef55fb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.033167 12452 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20240503 14:13:38.039985 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 3b03512775d74ec9883ae6404eae58a3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.040206 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 3b03512775d74ec9883ae6404eae58a3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.040405 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b03512775d74ec9883ae6404eae58a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.040629 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b03512775d74ec9883ae6404eae58a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.042487 9909 tablet_service.cc:1467] Processing CreateTablet for tablet f2f54c0c48b848ed898534efef55fb62 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.042896 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f54c0c48b848ed898534efef55fb62. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:13:38.043203 6866 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b154f780fc742fa9b0efd609c307f95. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:38.044000 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 3b03512775d74ec9883ae6404eae58a3 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.044366 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b03512775d74ec9883ae6404eae58a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.048002 6910 tablet_service.cc:1467] Processing CreateTablet for tablet f2f54c0c48b848ed898534efef55fb62 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=5f9fd6d68cd4456e887025f13c0172b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:38.048424 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f54c0c48b848ed898534efef55fb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.060875 12515 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:38.062762 12515 tablet_bootstrap.cc:654] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.065749 12515 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:38.065994 12515 ts_tablet_manager.cc:1367] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.005s
I20240503 14:13:38.067404 12515 raft_consensus.cc:348] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.067898 12515 raft_consensus.cc:374] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.068053 12515 raft_consensus.cc:729] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.068360 12515 consensus_queue.cc:260] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.069509 12515 ts_tablet_manager.cc:1398] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:38.070816 12517 tablet_bootstrap.cc:492] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:38.072414 12517 tablet_bootstrap.cc:654] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.080041 12449 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:38.080291 6655 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:38.081498 12449 tablet_bootstrap.cc:654] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.082633 12519 tablet_bootstrap.cc:492] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:38.085261 12519 tablet_bootstrap.cc:654] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.087142 12522 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:38.088241 12517 tablet_bootstrap.cc:492] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:38.088490 12517 ts_tablet_manager.cc:1367] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.006s
I20240503 14:13:38.089555 12517 raft_consensus.cc:348] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.089933 12517 raft_consensus.cc:374] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.090058 12517 raft_consensus.cc:729] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.093729 12519 tablet_bootstrap.cc:492] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:38.095026 12519 ts_tablet_manager.cc:1367] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:13:38.095785 12452 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:38.097088 12519 raft_consensus.cc:348] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.098362 12522 tablet_bootstrap.cc:654] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.098587 12519 raft_consensus.cc:374] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.098807 12519 raft_consensus.cc:729] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.099054 12515 tablet_bootstrap.cc:492] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:38.099220 12519 consensus_queue.cc:260] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.100811 12515 tablet_bootstrap.cc:654] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.102488 12494 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:38.103112 12494 tablet.cc:1620] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:38.105142 12495 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:38.105764 12495 tablet.cc:1620] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:38.107782 12519 ts_tablet_manager.cc:1398] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:13:38.113375 12522 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:38.114183 12522 ts_tablet_manager.cc:1367] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240503 14:13:38.115734 12496 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:38.116277 12496 tablet.cc:1620] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: MemRowSet was empty: no flush needed.
I20240503 14:13:38.116338 12515 tablet_bootstrap.cc:492] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:38.116569 12515 ts_tablet_manager.cc:1367] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20240503 14:13:38.116732 12522 raft_consensus.cc:348] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.117503 12515 raft_consensus.cc:348] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.117650 12522 raft_consensus.cc:374] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.117811 12515 raft_consensus.cc:374] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.117875 12522 raft_consensus.cc:729] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.117962 12515 raft_consensus.cc:729] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.118356 12515 consensus_queue.cc:260] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.118366 12522 consensus_queue.cc:260] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.119395 12515 ts_tablet_manager.cc:1398] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:38.122823 9664 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:38.124703 6653 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 6 to 7, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.090721 12517 consensus_queue.cc:260] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.128058 12517 ts_tablet_manager.cc:1398] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20240503 14:13:38.129915 12523 tablet_bootstrap.cc:492] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:38.131937 12523 tablet_bootstrap.cc:654] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.133266 6655 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=6d2610fcedab4c658cce90042c623b80] alter complete (version 2)
I20240503 14:13:38.138710 12449 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:38.138882 12449 ts_tablet_manager.cc:1367] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.004s
I20240503 14:13:38.139575 12449 raft_consensus.cc:348] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.139808 12449 raft_consensus.cc:374] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.139871 12449 raft_consensus.cc:729] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.140077 12449 consensus_queue.cc:260] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.140591 12449 ts_tablet_manager.cc:1398] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:38.142426 12523 tablet_bootstrap.cc:492] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:38.142693 12523 ts_tablet_manager.cc:1367] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240503 14:13:38.143666 12523 raft_consensus.cc:348] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.143956 12523 raft_consensus.cc:374] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.144070 12523 raft_consensus.cc:729] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.144371 12523 consensus_queue.cc:260] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.147562 12522 ts_tablet_manager.cc:1398] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240503 14:13:38.149462 12518 tablet_bootstrap.cc:492] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:38.151495 12523 ts_tablet_manager.cc:1398] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:38.171051 12518 tablet_bootstrap.cc:654] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.171737 12448 raft_consensus.cc:482] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.171905 12448 raft_consensus.cc:504] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.172600 12448 leader_election.cc:288] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:38.172531 7188 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:38.173614 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d8cb8f47304625999bb9a188001629" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
W20240503 14:13:38.174307 7127 leader_election.cc:341] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:38.174765 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d8cb8f47304625999bb9a188001629" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:38.174978 9928 raft_consensus.cc:2454] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:38.175479 7125 leader_election.cc:302] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, ea801601713a4d55aee0de9e3473e0bc; no voters: 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:38.176164 12448 raft_consensus.cc:2781] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.176244 12448 raft_consensus.cc:482] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.176313 12448 raft_consensus.cc:3037] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.179262 12448 raft_consensus.cc:504] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.181023 12518 tablet_bootstrap.cc:492] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:38.181191 12518 ts_tablet_manager.cc:1367] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.003s
I20240503 14:13:38.182050 12518 raft_consensus.cc:348] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.182256 12518 raft_consensus.cc:374] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.182338 12518 raft_consensus.cc:729] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.182572 12518 consensus_queue.cc:260] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.183193 12518 ts_tablet_manager.cc:1398] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:38.183687 12518 tablet_bootstrap.cc:492] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:38.185892 12518 tablet_bootstrap.cc:654] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.191993 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d8cb8f47304625999bb9a188001629" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:38.192227 9928 raft_consensus.cc:3037] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.194924 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d8cb8f47304625999bb9a188001629" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:38.195123 9928 raft_consensus.cc:2454] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:38.195451 7125 leader_election.cc:302] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:38.195793 12412 raft_consensus.cc:2781] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.195940 12412 raft_consensus.cc:686] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:38.196192 12412 consensus_queue.cc:237] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.196389 12518 tablet_bootstrap.cc:492] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:38.196547 12518 ts_tablet_manager.cc:1367] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.009s
W20240503 14:13:38.196575 7127 leader_election.cc:341] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:38.197235 12518 raft_consensus.cc:348] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.197326 12448 leader_election.cc:288] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:38.197455 12518 raft_consensus.cc:374] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.197535 12518 raft_consensus.cc:729] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.197755 12518 consensus_queue.cc:260] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.199944 12518 ts_tablet_manager.cc:1398] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:38.209124 6653 catalog_manager.cc:5484] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.218325 9797 consensus_queue.cc:237] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:38.222195 9664 raft_consensus.cc:1261] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:38.222321 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13f41f236987487b8bca287c7ec1fdd3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:38.222873 12458 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:38.222997 9928 raft_consensus.cc:1261] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.223640 12458 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:38.225804 12453 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:38.226475 9664 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:38.233533 6653 catalog_manager.cc:5484] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:38.239239 12015 raft_consensus.cc:482] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.239436 12015 raft_consensus.cc:504] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.240320 12015 leader_election.cc:288] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:38.241299 7188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d8cb8f47304625999bb9a188001629" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:38.241860 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d8cb8f47304625999bb9a188001629" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:38.242125 9930 raft_consensus.cc:2379] T 27d8cb8f47304625999bb9a188001629 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b7e36bf354f480c90859cb8867b59cf in current term 1: Already voted for candidate e1bf5fb7402c46829974e2f74cd88708 in this term.
I20240503 14:13:38.241784 9928 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:38.242635 9468 leader_election.cc:302] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: e1bf5fb7402c46829974e2f74cd88708, ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:38.243225 12015 raft_consensus.cc:3037] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.250270 12015 raft_consensus.cc:2726] T 27d8cb8f47304625999bb9a188001629 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:38.272585 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:38.291270 12494 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:38.295046 12496 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:38.296241 12495 tablet.cc:1722] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:38.302688 6657 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=6d2610fcedab4c658cce90042c623b80] alter complete (version 3)
I20240503 14:13:38.318615 12341 raft_consensus.cc:482] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.318794 12341 raft_consensus.cc:504] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.319798 12341 leader_election.cc:288] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:38.321516 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b03512775d74ec9883ae6404eae58a3" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:38.321538 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b03512775d74ec9883ae6404eae58a3" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:38.321765 7060 raft_consensus.cc:2454] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:38.321784 9531 raft_consensus.cc:2454] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:38.322912 6736 leader_election.cc:302] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:38.324373 12341 raft_consensus.cc:2781] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.324466 12341 raft_consensus.cc:482] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.324535 12341 raft_consensus.cc:3037] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.326612 12341 raft_consensus.cc:504] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.327134 12341 leader_election.cc:288] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:38.327551 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b03512775d74ec9883ae6404eae58a3" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:38.327661 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b03512775d74ec9883ae6404eae58a3" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:38.327769 7060 raft_consensus.cc:3037] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.327885 9531 raft_consensus.cc:3037] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.330545 9531 raft_consensus.cc:2454] T 3b03512775d74ec9883ae6404eae58a3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:38.331801 6736 leader_election.cc:302] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:38.332093 12341 raft_consensus.cc:2781] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.332218 12341 raft_consensus.cc:686] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:38.332466 12341 consensus_queue.cc:237] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:38.338907 7060 raft_consensus.cc:2454] T 3b03512775d74ec9883ae6404eae58a3 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:38.338073 9531 consensus_queue.cc:237] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:38.344403 6653 catalog_manager.cc:5484] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.361287 12549 ts_tablet_manager.cc:902] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:38.370692 6929 raft_consensus.cc:1261] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:38.371381 9468 consensus_peers.cc:485] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4b86d126b92400f9b14a9e4fb527fc9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:38.372409 9928 raft_consensus.cc:1261] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.373203 12459 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:38.374560 12459 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:38.393427 12249 raft_consensus.cc:482] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.393684 12249 raft_consensus.cc:504] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.394924 12249 leader_election.cc:288] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:38.396059 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f54c0c48b848ed898534efef55fb62" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:38.396395 9930 raft_consensus.cc:2454] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:38.396963 9599 leader_election.cc:302] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:38.397804 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f54c0c48b848ed898534efef55fb62" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:38.398102 6930 raft_consensus.cc:2454] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:38.399037 12249 raft_consensus.cc:2781] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.399185 12249 raft_consensus.cc:482] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.399288 12249 raft_consensus.cc:3037] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.403380 12249 raft_consensus.cc:504] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.405257 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f54c0c48b848ed898534efef55fb62" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:38.405488 6930 raft_consensus.cc:3037] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.406186 12459 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:38.409404 9930 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:38.413753 6653 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:38.414057 6653 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:38.414477 6930 raft_consensus.cc:2454] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:38.415336 9599 leader_election.cc:302] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:38.415767 12541 raft_consensus.cc:2781] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.415944 12541 raft_consensus.cc:686] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:38.416321 12541 consensus_queue.cc:237] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:38.417300 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f54c0c48b848ed898534efef55fb62" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:38.417577 9930 raft_consensus.cc:3037] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.418972 12249 leader_election.cc:288] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:38.420953 9930 raft_consensus.cc:2454] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:38.420883 6655 catalog_manager.cc:5484] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:38.421676 6653 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 4e7db2cf1f6a4d62bf5562125591fb51
I20240503 14:13:38.423825 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7db2cf1f6a4d62bf5562125591fb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:38 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:38.424391 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7db2cf1f6a4d62bf5562125591fb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:38 UTC) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:38.424418 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e7db2cf1f6a4d62bf5562125591fb51 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:38 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:38.425024 12561 tablet_replica.cc:330] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:38.426817 12561 raft_consensus.cc:2229] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:38.427214 12561 raft_consensus.cc:2258] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:38.433481 12549 tablet_copy_client.cc:308] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:38.434367 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 89ff0991a4e245609405ff945388417b from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:47571)
I20240503 14:13:38.434549 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 89ff0991a4e245609405ff945388417b from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:47571: session id = f1a669e28f014fe287932acf56bebc13-89ff0991a4e245609405ff945388417b
I20240503 14:13:38.437012 12559 tablet_replica.cc:330] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:38.437309 12559 raft_consensus.cc:2229] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:38.437594 12559 raft_consensus.cc:2258] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:38.438309 9819 tablet_copy_source_session.cc:205] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.438834 12559 ts_tablet_manager.cc:1875] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:38.440588 12549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89ff0991a4e245609405ff945388417b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.448779 12559 ts_tablet_manager.cc:1888] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:38.449098 12559 log.cc:1196] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/4e7db2cf1f6a4d62bf5562125591fb51
I20240503 14:13:38.449189 12549 tablet_copy_client.cc:783] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.449523 12549 tablet_copy_client.cc:647] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.449641 12559 ts_tablet_manager.cc:1909] T 4e7db2cf1f6a4d62bf5562125591fb51 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:38.451191 12549 tablet_copy_client.cc:523] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:38.451498 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 4e7db2cf1f6a4d62bf5562125591fb51 (table rebalancer_dynamic_table_new_name [id=6d2610fcedab4c658cce90042c623b80]) successfully deleted
I20240503 14:13:38.455334 12549 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:38.464016 6929 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:38.471335 12549 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:38.471928 12549 tablet_bootstrap.cc:492] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:38.472281 12549 ts_tablet_manager.cc:1367] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20240503 14:13:38.473101 12549 raft_consensus.cc:348] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:38.473671 12549 raft_consensus.cc:729] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:38.473956 12549 consensus_queue.cc:260] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:38.474200 12562 tablet_replica.cc:330] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:38.474467 12562 raft_consensus.cc:2229] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:38.474588 12549 ts_tablet_manager.cc:1398] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:38.474938 12562 raft_consensus.cc:2258] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:38.475347 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-89ff0991a4e245609405ff945388417b received from {username='slave'} at 127.0.186.1:47571
I20240503 14:13:38.475507 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session f1a669e28f014fe287932acf56bebc13-89ff0991a4e245609405ff945388417b on tablet 89ff0991a4e245609405ff945388417b with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:38.476109 12562 ts_tablet_manager.cc:1875] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:38.481333 12565 ts_tablet_manager.cc:902] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:38.483325 6657 catalog_manager.cc:5484] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.485505 12561 ts_tablet_manager.cc:1875] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:38.488555 12565 tablet_copy_client.cc:308] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:38.489315 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet b6d600a8b0134a018f8c9e5a1d9b79c5 from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:51299)
I20240503 14:13:38.489660 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet b6d600a8b0134a018f8c9e5a1d9b79c5 from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:51299: session id = e1bf5fb7402c46829974e2f74cd88708-b6d600a8b0134a018f8c9e5a1d9b79c5
I20240503 14:13:38.491951 12557 ts_tablet_manager.cc:902] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:38.492368 6951 tablet_copy_source_session.cc:205] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.494330 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d600a8b0134a018f8c9e5a1d9b79c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.501205 12565 tablet_copy_client.cc:783] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.501552 12565 tablet_copy_client.cc:647] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.503571 12565 tablet_copy_client.cc:523] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:38.507134 12565 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:38.509198 12562 ts_tablet_manager.cc:1888] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:38.509344 12562 log.cc:1196] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/4e7db2cf1f6a4d62bf5562125591fb51
I20240503 14:13:38.517262 12557 tablet_copy_client.cc:308] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:38.517807 12562 ts_tablet_manager.cc:1909] T 4e7db2cf1f6a4d62bf5562125591fb51 P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:38.518126 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet f6f8dfcb870a4461b03415db9c33452e from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:32947)
I20240503 14:13:38.518266 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet f6f8dfcb870a4461b03415db9c33452e from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:32947: session id = 09723f236d8044538d920a37b5d9aa8c-f6f8dfcb870a4461b03415db9c33452e
I20240503 14:13:38.518997 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 4e7db2cf1f6a4d62bf5562125591fb51 (table rebalancer_dynamic_table_new_name [id=6d2610fcedab4c658cce90042c623b80]) successfully deleted
I20240503 14:13:38.518846 12561 ts_tablet_manager.cc:1888] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:38.519362 12561 log.cc:1196] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/4e7db2cf1f6a4d62bf5562125591fb51
I20240503 14:13:38.519846 12561 ts_tablet_manager.cc:1909] T 4e7db2cf1f6a4d62bf5562125591fb51 P 7186ed8510424fd597173eaf10c57b12: Deleting consensus metadata
I20240503 14:13:38.521337 9950 tablet_copy_source_session.cc:205] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.522082 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 4e7db2cf1f6a4d62bf5562125591fb51 (table rebalancer_dynamic_table_new_name [id=6d2610fcedab4c658cce90042c623b80]) successfully deleted
I20240503 14:13:38.524489 9928 consensus_queue.cc:237] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:38.525576 12565 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:38.525578 12557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6f8dfcb870a4461b03415db9c33452e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.526167 12565 tablet_bootstrap.cc:492] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:38.526543 12565 ts_tablet_manager.cc:1367] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.015s
I20240503 14:13:38.527402 12565 raft_consensus.cc:348] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:38.527765 12565 raft_consensus.cc:729] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:38.528049 12565 consensus_queue.cc:260] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:38.528370 6657 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:38.528671 12565 ts_tablet_manager.cc:1398] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:38.529489 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-b6d600a8b0134a018f8c9e5a1d9b79c5 received from {username='slave'} at 127.0.186.4:51299
I20240503 14:13:38.529652 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-b6d600a8b0134a018f8c9e5a1d9b79c5 on tablet b6d600a8b0134a018f8c9e5a1d9b79c5 with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:38.538163 12557 tablet_copy_client.cc:783] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.539125 12557 tablet_copy_client.cc:647] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.539935 7188 raft_consensus.cc:1261] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:38.539945 9863 consensus_peers.cc:485] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 005dec40a9054f7fa95ea8722e754e24. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:38.540556 12313 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:38.541630 9531 raft_consensus.cc:1261] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.542546 12313 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:38.543639 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 4b09a64c5814437184354829e46072a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be]), partition=
I20240503 14:13:38.544034 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b09a64c5814437184354829e46072a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.544076 12313 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:38.544937 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 4b09a64c5814437184354829e46072a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be]), partition=
I20240503 14:13:38.545287 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b09a64c5814437184354829e46072a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.548362 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 4b09a64c5814437184354829e46072a1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be]), partition=
I20240503 14:13:38.548775 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b09a64c5814437184354829e46072a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.552389 7188 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:38.554874 12557 tablet_copy_client.cc:523] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:38.558393 12557 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:38.566848 6657 catalog_manager.cc:5484] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:38.571705 12523 tablet_bootstrap.cc:492] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:38.573746 12523 tablet_bootstrap.cc:654] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.574779 12515 tablet_bootstrap.cc:492] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:38.576390 12515 tablet_bootstrap.cc:654] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.583112 12523 tablet_bootstrap.cc:492] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:38.583292 12523 ts_tablet_manager.cc:1367] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 14:13:38.588758 12523 raft_consensus.cc:348] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.589175 12523 raft_consensus.cc:374] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.589329 12523 raft_consensus.cc:729] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.589653 12523 consensus_queue.cc:260] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.594074 12523 ts_tablet_manager.cc:1398] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20240503 14:13:38.608676 9531 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:38.614100 12588 ts_tablet_manager.cc:902] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:38.617815 12515 tablet_bootstrap.cc:492] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:38.617993 12515 ts_tablet_manager.cc:1367] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s
I20240503 14:13:38.618885 12515 raft_consensus.cc:348] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.619125 12515 raft_consensus.cc:374] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.619225 12515 raft_consensus.cc:729] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.619499 12515 consensus_queue.cc:260] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.620177 12515 ts_tablet_manager.cc:1398] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:38.635504 12557 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:38.636135 12557 tablet_bootstrap.cc:492] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:38.636510 12557 ts_tablet_manager.cc:1367] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.001s
I20240503 14:13:38.637351 12583 tablet_bootstrap.cc:492] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:38.640255 12583 tablet_bootstrap.cc:654] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:38.638398 12557 raft_consensus.cc:348] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:38.642066 12557 raft_consensus.cc:729] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:38.642619 12557 consensus_queue.cc:260] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:38.647979 12557 ts_tablet_manager.cc:1398] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:13:38.652357 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-f6f8dfcb870a4461b03415db9c33452e received from {username='slave'} at 127.0.186.3:32947
I20240503 14:13:38.652539 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-f6f8dfcb870a4461b03415db9c33452e on tablet f6f8dfcb870a4461b03415db9c33452e with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:38.655223 12590 ts_tablet_manager.cc:902] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:38.679800 12588 tablet_copy_client.cc:308] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:38.680541 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 13f41f236987487b8bca287c7ec1fdd3 from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:46691)
I20240503 14:13:38.681249 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 13f41f236987487b8bca287c7ec1fdd3 from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:46691: session id = 09723f236d8044538d920a37b5d9aa8c-13f41f236987487b8bca287c7ec1fdd3
I20240503 14:13:38.681756 12583 tablet_bootstrap.cc:492] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:38.681910 12583 ts_tablet_manager.cc:1367] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.003s
I20240503 14:13:38.681720 7060 consensus_queue.cc:237] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:38.682828 12583 raft_consensus.cc:348] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.683071 12583 raft_consensus.cc:374] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:38.683174 12583 raft_consensus.cc:729] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:38.683420 12583 consensus_queue.cc:260] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.684424 9819 tablet_copy_source_session.cc:205] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.686983 9662 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:38.687611 12460 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:38.687916 6929 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:38.688931 12460 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:13:38.692392 6996 consensus_peers.cc:485] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a44ed92d65114bfda2e0f38054b17ada. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:38.697495 12583 ts_tablet_manager.cc:1398] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 14:13:38.702419 12590 tablet_copy_client.cc:308] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:38.703133 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 9b154f780fc742fa9b0efd609c307f95 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:58905)
I20240503 14:13:38.703297 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 9b154f780fc742fa9b0efd609c307f95 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:58905: session id = 3efdbd3506274dba846ae40404f0572c-9b154f780fc742fa9b0efd609c307f95
I20240503 14:13:38.706286 6951 tablet_copy_source_session.cc:205] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.709152 12590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b154f780fc742fa9b0efd609c307f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.711467 12588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13f41f236987487b8bca287c7ec1fdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.717494 12590 tablet_copy_client.cc:783] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.718444 12590 tablet_copy_client.cc:647] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.720239 12590 tablet_copy_client.cc:523] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:38.722329 10371 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:38.723217 6929 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:38.724248 12590 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:38.726428 12588 tablet_copy_client.cc:783] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.727901 12588 tablet_copy_client.cc:647] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.730578 6653 catalog_manager.cc:5484] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 6 to 7, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:38.731115 12588 tablet_copy_client.cc:523] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:38.736250 12588 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:38.751813 9664 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:38.769887 12588 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:38.770258 12588 tablet_bootstrap.cc:492] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:38.770499 12588 ts_tablet_manager.cc:1367] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.007s
I20240503 14:13:38.771049 12588 raft_consensus.cc:348] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:38.771242 12588 raft_consensus.cc:729] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:38.771442 12588 consensus_queue.cc:260] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:38.771962 12588 ts_tablet_manager.cc:1398] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240503 14:13:38.772989 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-13f41f236987487b8bca287c7ec1fdd3 received from {username='slave'} at 127.0.186.3:46691
I20240503 14:13:38.773123 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-13f41f236987487b8bca287c7ec1fdd3 on tablet 13f41f236987487b8bca287c7ec1fdd3 with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:38.777482 11977 raft_consensus.cc:482] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:38.777683 11977 raft_consensus.cc:504] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.778344 11977 leader_election.cc:288] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:38.779023 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b09a64c5814437184354829e46072a1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:38.779225 7060 raft_consensus.cc:2454] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:38.780026 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b09a64c5814437184354829e46072a1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:38.780241 9797 raft_consensus.cc:2454] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:38.781447 9865 leader_election.cc:302] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:38.783691 12411 consensus_queue.cc:1035] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.786821 11977 raft_consensus.cc:2781] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:38.786967 11977 raft_consensus.cc:482] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:38.787040 11977 raft_consensus.cc:3037] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.791149 11977 raft_consensus.cc:504] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.792657 7060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b09a64c5814437184354829e46072a1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:38.792865 7060 raft_consensus.cc:3037] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.797398 7060 raft_consensus.cc:2454] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:38.797940 9865 leader_election.cc:302] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:38.798267 12313 raft_consensus.cc:2781] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:38.798398 12313 raft_consensus.cc:686] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:38.798760 12313 consensus_queue.cc:237] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:38.799685 11977 leader_election.cc:288] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:38.801031 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b09a64c5814437184354829e46072a1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:38.801265 9797 raft_consensus.cc:3037] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:38.804556 9797 raft_consensus.cc:2454] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:38.815369 12448 consensus_queue.cc:1035] T 27d8cb8f47304625999bb9a188001629 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.825486 6653 catalog_manager.cc:5484] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.835523 12341 consensus_queue.cc:1035] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.840502 12341 consensus_queue.cc:1035] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.885843 12565 ts_tablet_manager.cc:902] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:38.894558 12590 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:38.895341 12590 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:38.895787 12590 ts_tablet_manager.cc:1367] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.172s user 0.056s sys 0.011s
I20240503 14:13:38.896936 12590 raft_consensus.cc:348] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:38.899384 12590 raft_consensus.cc:729] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:38.899775 12590 consensus_queue.cc:260] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:38.905792 12590 ts_tablet_manager.cc:1398] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 14:13:38.930233 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-9b154f780fc742fa9b0efd609c307f95 received from {username='slave'} at 127.0.186.7:58905
I20240503 14:13:38.930393 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-9b154f780fc742fa9b0efd609c307f95 on tablet 9b154f780fc742fa9b0efd609c307f95 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:38.943265 12565 tablet_copy_client.cc:308] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.5:36609
I20240503 14:13:38.944362 9553 tablet_copy_service.cc:140] P 5b7e36bf354f480c90859cb8867b59cf: Received BeginTabletCopySession request for tablet f4b86d126b92400f9b14a9e4fb527fc9 from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:59313)
I20240503 14:13:38.944598 9553 tablet_copy_service.cc:161] P 5b7e36bf354f480c90859cb8867b59cf: Beginning new tablet copy session on tablet f4b86d126b92400f9b14a9e4fb527fc9 from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:59313: session id = e1bf5fb7402c46829974e2f74cd88708-f4b86d126b92400f9b14a9e4fb527fc9
I20240503 14:13:38.948689 9553 tablet_copy_source_session.cc:205] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:38.950462 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4b86d126b92400f9b14a9e4fb527fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:38.952890 9797 consensus_queue.cc:237] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:38.956601 12565 tablet_copy_client.cc:783] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:38.956924 12565 tablet_copy_client.cc:647] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:38.958750 12565 tablet_copy_client.cc:523] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:13:38.961237 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58548735201648fa805fdf6af390c088. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:38.964181 12565 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:38.966573 9531 raft_consensus.cc:1261] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.967461 12454 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:38.970019 9663 raft_consensus.cc:1261] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.970346 6800 raft_consensus.cc:1203] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:38.971906 12634 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:38.972440 6655 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:38.972657 12453 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:38.974329 12565 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:38.974721 12565 tablet_bootstrap.cc:492] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:38.974960 12565 ts_tablet_manager.cc:1367] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240503 14:13:38.976506 12565 raft_consensus.cc:348] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:38.976866 12565 raft_consensus.cc:729] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:38.977102 12565 consensus_queue.cc:260] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:38.979574 12565 ts_tablet_manager.cc:1398] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 14:13:38.980587 9553 tablet_copy_service.cc:342] P 5b7e36bf354f480c90859cb8867b59cf: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-f4b86d126b92400f9b14a9e4fb527fc9 received from {username='slave'} at 127.0.186.4:59313
I20240503 14:13:38.980748 9553 tablet_copy_service.cc:433] P 5b7e36bf354f480c90859cb8867b59cf: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-f4b86d126b92400f9b14a9e4fb527fc9 on tablet f4b86d126b92400f9b14a9e4fb527fc9 with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:38.981199 9797 raft_consensus.cc:1261] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.981667 7058 raft_consensus.cc:1261] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:38.981719 9663 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:38.984026 11977 consensus_queue.cc:1035] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.984290 12292 consensus_queue.cc:1035] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.986348 6657 catalog_manager.cc:5484] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:38.987305 12249 consensus_queue.cc:1035] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:38.987080 9533 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:38.989817 7060 raft_consensus.cc:1203] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:38.996628 7060 raft_consensus.cc:1203] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:39.015801 6657 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 4b09a64c5814437184354829e46072a1 (table rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be]). Expected version 1 got 0
I20240503 14:13:39.019061 12646 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:39.025727 12646 tablet.cc:1620] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:39.031862 12249 consensus_queue.cc:1035] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.034854 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:39.024955 6657 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 4b09a64c5814437184354829e46072a1 (table rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be]). Expected version 1 got 0
I20240503 14:13:39.036438 6655 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 4b09a64c5814437184354829e46072a1 (table rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be]). Expected version 1 got 0
I20240503 14:13:39.043423 12644 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:39.043999 12644 tablet.cc:1620] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:39.044785 12649 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:39.045459 6655 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be] alter complete (version 1)
I20240503 14:13:39.047511 6655 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 4b09a64c5814437184354829e46072a1 (table rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be]). Expected version 1 got 0
I20240503 14:13:39.048246 12649 tablet.cc:1620] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:39.050333 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 2f8a6c04df3443c0af8e4f20fcd171ff (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.050908 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f8a6c04df3443c0af8e4f20fcd171ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.051259 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3b35a8172d478ba2b4717569ee993e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.051750 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3b35a8172d478ba2b4717569ee993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.053099 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 5895d998bfb3439386fcbc2c74f8d9a6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.053543 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5895d998bfb3439386fcbc2c74f8d9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.055225 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3b35a8172d478ba2b4717569ee993e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.055711 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3b35a8172d478ba2b4717569ee993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.060144 12523 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:39.060704 6911 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3b35a8172d478ba2b4717569ee993e (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.061144 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3b35a8172d478ba2b4717569ee993e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.061822 12523 tablet_bootstrap.cc:654] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.062919 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 2f8a6c04df3443c0af8e4f20fcd171ff (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.063211 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f8a6c04df3443c0af8e4f20fcd171ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.063777 12515 tablet_bootstrap.cc:492] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:39.064021 6909 tablet_service.cc:1467] Processing CreateTablet for tablet 5895d998bfb3439386fcbc2c74f8d9a6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.064378 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5895d998bfb3439386fcbc2c74f8d9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.066183 12515 tablet_bootstrap.cc:654] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.068096 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 2f8a6c04df3443c0af8e4f20fcd171ff (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.068557 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f8a6c04df3443c0af8e4f20fcd171ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.075614 12523 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:39.075785 12523 ts_tablet_manager.cc:1367] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20240503 14:13:39.078541 12523 raft_consensus.cc:348] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.078836 12523 raft_consensus.cc:374] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.078955 12523 raft_consensus.cc:729] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.079236 12523 consensus_queue.cc:260] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.079793 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 5895d998bfb3439386fcbc2c74f8d9a6 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:39.080078 12523 ts_tablet_manager.cc:1398] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:39.080605 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5895d998bfb3439386fcbc2c74f8d9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.108634 7188 raft_consensus.cc:1203] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:39.114830 12660 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:39.120126 12660 tablet_bootstrap.cc:654] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.128022 12661 ts_tablet_manager.cc:902] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:39.132484 9930 consensus_queue.cc:237] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:39.132776 12660 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:39.133361 12660 ts_tablet_manager.cc:1367] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.004s
I20240503 14:13:39.134609 12660 raft_consensus.cc:348] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.135874 12660 raft_consensus.cc:374] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.136190 12660 raft_consensus.cc:729] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.136565 12660 consensus_queue.cc:260] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.140240 12663 tablet_bootstrap.cc:492] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:39.142645 12660 ts_tablet_manager.cc:1398] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240503 14:13:39.147008 12660 tablet_bootstrap.cc:492] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:39.148949 12660 tablet_bootstrap.cc:654] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.151516 10371 raft_consensus.cc:482] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.151764 10371 raft_consensus.cc:504] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.152827 10371 leader_election.cc:288] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
W20240503 14:13:39.153754 9864 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b97ddea52b624c668cb0b28ab0f38196. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:39.154368 6930 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:39.154794 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:39.155112 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:39.155118 12314 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:39.155640 6995 leader_election.cc:341] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:39.155716 9533 raft_consensus.cc:2454] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:39.156117 6997 leader_election.cc:302] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf; no voters: 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:39.156729 10371 raft_consensus.cc:2781] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.156836 10371 raft_consensus.cc:482] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.156924 10371 raft_consensus.cc:3037] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.157359 12590 ts_tablet_manager.cc:902] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:39.161351 12660 tablet_bootstrap.cc:492] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:39.162009 12660 ts_tablet_manager.cc:1367] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240503 14:13:39.162983 12660 raft_consensus.cc:348] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.164130 12660 raft_consensus.cc:374] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.164356 12660 raft_consensus.cc:729] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.164832 12660 consensus_queue.cc:260] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.164963 10371 raft_consensus.cc:504] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.167898 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
W20240503 14:13:39.168491 6995 leader_election.cc:341] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:39.169951 12660 ts_tablet_manager.cc:1398] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:13:39.170694 10371 leader_election.cc:288] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:39.175513 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:39.175849 9533 raft_consensus.cc:3037] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.177866 6655 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:39.187052 12666 tablet_bootstrap.cc:492] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:39.188894 12644 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:39.190349 12646 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:39.190829 12646 tablet.cc:1620] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:39.190150 9533 raft_consensus.cc:2454] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:39.192028 12644 tablet.cc:1620] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:39.192157 6997 leader_election.cc:302] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf; no voters: 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:39.192657 10371 raft_consensus.cc:2781] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.193186 10371 raft_consensus.cc:686] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:39.193948 10371 consensus_queue.cc:237] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.204126 6655 catalog_manager.cc:5576] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783) does not have the latest schema for tablet 4b09a64c5814437184354829e46072a1 (table rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be]). Expected version 2 got 1
I20240503 14:13:39.212541 9797 raft_consensus.cc:1203] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:39.215068 6651 catalog_manager.cc:5484] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:39.217271 12663 tablet_bootstrap.cc:654] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.224357 9663 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:39.227485 11977 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:39.229912 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=9ffc1e1686fb45f487675af12101b8be] alter complete (version 2)
I20240503 14:13:39.230212 12666 tablet_bootstrap.cc:654] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.231791 12515 tablet_bootstrap.cc:492] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:39.232031 12515 ts_tablet_manager.cc:1367] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.168s user 0.009s sys 0.005s
I20240503 14:13:39.232832 12515 raft_consensus.cc:348] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.233094 12515 raft_consensus.cc:374] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.233227 12515 raft_consensus.cc:729] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.233843 12515 consensus_queue.cc:260] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.235340 12515 ts_tablet_manager.cc:1398] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:39.238215 12314 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:39.238803 6931 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:39.243319 12663 tablet_bootstrap.cc:492] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:39.243553 12663 ts_tablet_manager.cc:1367] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.000s
I20240503 14:13:39.244688 12663 raft_consensus.cc:348] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.246958 12663 raft_consensus.cc:374] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.247084 12663 raft_consensus.cc:729] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.248518 6657 catalog_manager.cc:5484] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:39.248937 12649 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:39.249559 12649 tablet.cc:1620] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:39.250084 12663 consensus_queue.cc:260] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.255445 12663 ts_tablet_manager.cc:1398] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:39.261200 12313 raft_consensus.cc:482] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.261534 12666 tablet_bootstrap.cc:492] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:39.261451 12313 raft_consensus.cc:504] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.261704 12666 ts_tablet_manager.cc:1367] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.000s
I20240503 14:13:39.262332 12313 leader_election.cc:288] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:39.262594 12666 raft_consensus.cc:348] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.262807 12666 raft_consensus.cc:374] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.262902 12666 raft_consensus.cc:729] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.263113 12666 consensus_queue.cc:260] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.263271 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a6c04df3443c0af8e4f20fcd171ff" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
W20240503 14:13:39.263914 9863 leader_election.cc:341] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:39.267355 12590 tablet_copy_client.cc:308] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:39.268198 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet a44ed92d65114bfda2e0f38054b17ada from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:58913)
I20240503 14:13:39.268370 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet a44ed92d65114bfda2e0f38054b17ada from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:58913: session id = 3efdbd3506274dba846ae40404f0572c-a44ed92d65114bfda2e0f38054b17ada
I20240503 14:13:39.271446 7081 tablet_copy_source_session.cc:205] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:39.274122 12652 tablet_bootstrap.cc:492] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:39.274997 12590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a44ed92d65114bfda2e0f38054b17ada. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.279210 12652 tablet_bootstrap.cc:654] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.280714 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a6c04df3443c0af8e4f20fcd171ff" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:39.280987 9663 raft_consensus.cc:2454] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:39.283192 9863 leader_election.cc:302] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters: 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:39.285323 12292 raft_consensus.cc:2781] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.285465 12292 raft_consensus.cc:482] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.285560 12292 raft_consensus.cc:3037] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.289359 9663 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:39.290791 12666 ts_tablet_manager.cc:1398] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240503 14:13:39.293354 12292 raft_consensus.cc:504] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.294271 12292 leader_election.cc:288] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:39.294816 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a6c04df3443c0af8e4f20fcd171ff" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
W20240503 14:13:39.295853 9863 leader_election.cc:341] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:39.300153 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a6c04df3443c0af8e4f20fcd171ff" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:39.300446 9663 raft_consensus.cc:3037] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.305264 9663 raft_consensus.cc:2454] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:39.306900 9863 leader_election.cc:302] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters: 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:39.307404 12292 raft_consensus.cc:2781] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.307601 12292 raft_consensus.cc:686] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:39.307927 12292 consensus_queue.cc:237] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.308514 12661 tablet_copy_client.cc:308] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:39.313333 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet 005dec40a9054f7fa95ea8722e754e24 from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:40149)
I20240503 14:13:39.313521 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet 005dec40a9054f7fa95ea8722e754e24 from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:40149: session id = f1a669e28f014fe287932acf56bebc13-005dec40a9054f7fa95ea8722e754e24
I20240503 14:13:39.317502 9950 tablet_copy_source_session.cc:205] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:39.321460 12590 tablet_copy_client.cc:783] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:39.321765 12590 tablet_copy_client.cc:647] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:39.321538 6655 catalog_manager.cc:5484] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:39.326800 12661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 005dec40a9054f7fa95ea8722e754e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.327989 12590 tablet_copy_client.cc:523] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:39.332906 6655 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:39.333866 12590 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:39.341019 6798 consensus_queue.cc:237] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:39.343364 12644 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:39.344112 12646 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:39.352497 9533 raft_consensus.cc:1261] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:39.353216 12630 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:39.357951 12649 tablet.cc:1722] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:39.359927 12327 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:39.360596 9531 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
W20240503 14:13:39.363570 6734 consensus_peers.cc:485] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1949c165716e42cf89d542e6b9b7ae71. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:39.368275 6651 catalog_manager.cc:5484] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:39.372611 9797 raft_consensus.cc:1261] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:39.373265 12630 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:39.379195 6657 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=9ffc1e1686fb45f487675af12101b8be] alter complete (version 3)
I20240503 14:13:39.387557 12661 tablet_copy_client.cc:783] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:39.388175 12661 tablet_copy_client.cc:647] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:39.390307 12661 tablet_copy_client.cc:523] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:39.393579 12652 tablet_bootstrap.cc:492] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:39.393879 12652 ts_tablet_manager.cc:1367] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.120s user 0.008s sys 0.000s
I20240503 14:13:39.395090 12652 raft_consensus.cc:348] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.395532 12652 raft_consensus.cc:374] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.395740 12652 raft_consensus.cc:729] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.396193 12652 consensus_queue.cc:260] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.401355 12652 ts_tablet_manager.cc:1398] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:39.402602 12652 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:39.402884 9797 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:39.421195 12454 raft_consensus.cc:1050] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:39.422297 12454 consensus_queue.cc:237] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:39.424206 12652 tablet_bootstrap.cc:654] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.434826 12661 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:39.435429 9930 raft_consensus.cc:1261] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.438556 7188 raft_consensus.cc:1203] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:39.438803 12634 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.447568 6798 raft_consensus.cc:1261] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.447666 7060 raft_consensus.cc:1261] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.448316 12633 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.450708 12633 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.464033 12633 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:39.469866 9929 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:39.470430 12015 raft_consensus.cc:482] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.470594 12015 raft_consensus.cc:504] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.471387 12015 leader_election.cc:288] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:39.472146 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3b35a8172d478ba2b4717569ee993e" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:39.472153 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3b35a8172d478ba2b4717569ee993e" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:39.472365 6800 raft_consensus.cc:2454] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
W20240503 14:13:39.472680 9467 leader_election.cc:341] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:39.472887 9467 leader_election.cc:302] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters: 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:39.473143 12015 raft_consensus.cc:2781] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:39.473258 12015 raft_consensus.cc:482] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:39.473363 12015 raft_consensus.cc:3037] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.476670 6655 catalog_manager.cc:5484] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:39.488910 12015 raft_consensus.cc:504] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.489137 9531 consensus_queue.cc:237] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:39.487064 6798 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:39.493512 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3b35a8172d478ba2b4717569ee993e" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
W20240503 14:13:39.494246 9467 leader_election.cc:341] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:39.501318 12661 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:39.501739 12661 tablet_bootstrap.cc:492] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:39.501974 12661 ts_tablet_manager.cc:1367] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.067s user 0.011s sys 0.004s
I20240503 14:13:39.502686 12661 raft_consensus.cc:348] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:39.501485 9929 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:39.502923 12661 raft_consensus.cc:729] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:39.503106 12661 consensus_queue.cc:260] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:39.505033 7058 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:39.508350 12642 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:39.513206 6658 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:39.513336 9797 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:39.513451 6658 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:39.518854 12634 raft_consensus.cc:1050] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:39.519816 12634 consensus_queue.cc:237] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:39.529335 12661 ts_tablet_manager.cc:1398] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.027s user 0.002s sys 0.002s
I20240503 14:13:39.530160 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-005dec40a9054f7fa95ea8722e754e24 received from {username='slave'} at 127.0.186.1:40149
I20240503 14:13:39.530318 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session f1a669e28f014fe287932acf56bebc13-005dec40a9054f7fa95ea8722e754e24 on tablet 005dec40a9054f7fa95ea8722e754e24 with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:39.531350 12015 leader_election.cc:288] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:39.537642 12015 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:39.544582 12015 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:39.545403 9797 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:39.547817 6658 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 4b09a64c5814437184354829e46072a1
I20240503 14:13:39.549358 9798 consensus_queue.cc:237] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:39.551988 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b09a64c5814437184354829e46072a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:39 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:39.552889 7061 raft_consensus.cc:1261] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:39.553619 12633 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:39.558718 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b09a64c5814437184354829e46072a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:39 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:39.559721 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b09a64c5814437184354829e46072a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:39 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:39.565331 6798 raft_consensus.cc:1261] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:39.569813 12633 consensus_queue.cc:1035] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:39.573714 6657 catalog_manager.cc:5484] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 4 to 5, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:39.583315 12633 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:39.587841 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 89ff0991a4e245609405ff945388417b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:39.588526 12715 tablet_replica.cc:330] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:39.589537 6800 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:39.591413 12254 raft_consensus.cc:482] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:39.591768 12715 raft_consensus.cc:2229] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:39.591678 12254 raft_consensus.cc:504] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.592048 12715 raft_consensus.cc:2258] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:39.592626 12254 leader_election.cc:288] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:39.593019 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a6c04df3443c0af8e4f20fcd171ff" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:39.595520 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f8a6c04df3443c0af8e4f20fcd171ff" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:39.595995 9663 raft_consensus.cc:2379] T 2f8a6c04df3443c0af8e4f20fcd171ff P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7186ed8510424fd597173eaf10c57b12 in current term 1: Already voted for candidate ea801601713a4d55aee0de9e3473e0bc in this term.
I20240503 14:13:39.596381 12715 ts_tablet_manager.cc:1875] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:39.598048 6865 leader_election.cc:302] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12; no voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:39.599530 12254 raft_consensus.cc:3037] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.610800 6658 catalog_manager.cc:5484] T 89ff0991a4e245609405ff945388417b P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:39.615626 12313 raft_consensus.cc:1050] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:39.616523 12313 consensus_queue.cc:237] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:39.621040 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 89ff0991a4e245609405ff945388417b with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:39.622592 12590 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:39.623135 12590 tablet_bootstrap.cc:492] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:39.623430 12590 ts_tablet_manager.cc:1367] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.290s user 0.045s sys 0.013s
I20240503 14:13:39.623719 12254 raft_consensus.cc:2726] T 2f8a6c04df3443c0af8e4f20fcd171ff P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:39.624840 12715 ts_tablet_manager.cc:1888] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:39.630914 12714 tablet_replica.cc:330] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:39.631618 12714 raft_consensus.cc:2229] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:39.632009 12715 log.cc:1196] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/4b09a64c5814437184354829e46072a1
I20240503 14:13:39.632572 12714 raft_consensus.cc:2258] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:39.632802 7058 raft_consensus.cc:1261] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.633385 12313 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.633519 12715 ts_tablet_manager.cc:1909] T 4b09a64c5814437184354829e46072a1 P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
W20240503 14:13:39.633826 9467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 130533us
I20240503 14:13:39.636044 7060 raft_consensus.cc:1261] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.636592 12717 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.638443 9664 raft_consensus.cc:1261] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.638796 9663 raft_consensus.cc:1261] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.639190 12717 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.639566 6800 raft_consensus.cc:1261] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.640388 12313 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.640898 11977 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:13:39.641126 9467 consensus_peers.cc:485] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9641f7f5a3e4debb5aabc6b8a7497f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:39.641644 9929 raft_consensus.cc:1261] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.643399 12713 ts_tablet_manager.cc:902] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:39.644809 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3b35a8172d478ba2b4717569ee993e" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:39.645061 6798 raft_consensus.cc:3037] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:39.645879 12717 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.648048 7059 raft_consensus.cc:2932] T 89ff0991a4e245609405ff945388417b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:39.658191 6798 raft_consensus.cc:2454] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:39.662987 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 4b09a64c5814437184354829e46072a1 (table rebalancer_dynamic_table_new_name [id=9ffc1e1686fb45f487675af12101b8be]) successfully deleted
I20240503 14:13:39.662909 6653 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:39.664696 12590 raft_consensus.cc:348] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:39.666198 12590 raft_consensus.cc:729] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:39.666489 12590 consensus_queue.cc:260] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:39.667347 12590 ts_tablet_manager.cc:1398] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.044s user 0.005s sys 0.000s
I20240503 14:13:39.669209 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-a44ed92d65114bfda2e0f38054b17ada received from {username='slave'} at 127.0.186.7:58913
I20240503 14:13:39.669335 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-a44ed92d65114bfda2e0f38054b17ada on tablet a44ed92d65114bfda2e0f38054b17ada with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:39.682554 12654 raft_consensus.cc:1050] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
I20240503 14:13:39.683693 12654 consensus_queue.cc:237] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:39.686499 12313 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.691989 7057 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.697974 6798 raft_consensus.cc:1203] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:39.698817 6798 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.701617 12292 consensus_queue.cc:1035] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.712118 6651 catalog_manager.cc:5484] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.722505 9929 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:39.726733 9662 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.739944 12728 raft_consensus.cc:1050] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:39.740850 12728 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:39.746131 9798 raft_consensus.cc:1203] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:39.752909 9531 raft_consensus.cc:1261] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.753194 7190 raft_consensus.cc:1261] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.753906 12731 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.754482 12731 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.755649 9929 consensus_queue.cc:237] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:39.755852 9664 raft_consensus.cc:1261] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.756716 12731 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.758072 6798 raft_consensus.cc:1261] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:39.758702 12292 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:39.759101 12714 ts_tablet_manager.cc:1875] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:39.759922 7057 raft_consensus.cc:1261] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:39.761490 12313 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.762089 6798 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.765381 11977 consensus_queue.cc:1035] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:39.768011 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f6f8dfcb870a4461b03415db9c33452e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:39.768913 7057 raft_consensus.cc:2932] T f6f8dfcb870a4461b03415db9c33452e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.770172 6651 catalog_manager.cc:5484] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:39.771549 11977 consensus_queue.cc:1035] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:39.772572 6995 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:39.777292 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet f6f8dfcb870a4461b03415db9c33452e with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:39.781131 12652 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:39.783282 12652 ts_tablet_manager.cc:1367] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.381s user 0.049s sys 0.088s
I20240503 14:13:39.784448 12652 raft_consensus.cc:348] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.784822 12652 raft_consensus.cc:374] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.784914 12652 raft_consensus.cc:729] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.785190 12652 consensus_queue.cc:260] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:39.785936 12652 ts_tablet_manager.cc:1398] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:39.786294 12652 tablet_bootstrap.cc:492] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:39.788622 9467 leader_election.cc:302] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters: 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:39.788935 12015 raft_consensus.cc:2781] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:39.789448 12015 raft_consensus.cc:686] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:39.789880 12015 consensus_queue.cc:237] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.790314 12652 tablet_bootstrap.cc:654] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.795996 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 33058d2747b743bd97367234216a4694 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364]), partition=
I20240503 14:13:39.795998 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 33058d2747b743bd97367234216a4694 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364]), partition=
I20240503 14:13:39.796432 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33058d2747b743bd97367234216a4694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.796447 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33058d2747b743bd97367234216a4694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.799304 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 33058d2747b743bd97367234216a4694 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364]), partition=
I20240503 14:13:39.799749 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33058d2747b743bd97367234216a4694. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.808152 12712 tablet_replica.cc:330] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:39.808571 12712 raft_consensus.cc:2229] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:39.810849 12633 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.812163 12716 ts_tablet_manager.cc:902] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:39.818758 12254 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:39.821316 9665 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.814054 10371 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:39.818030 7057 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.828539 9533 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:39.829093 7190 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:39.830838 6651 catalog_manager.cc:5484] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.835440 12744 tablet_bootstrap.cc:492] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:39.836723 12712 raft_consensus.cc:2258] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:39.837883 12712 ts_tablet_manager.cc:1875] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:39.837962 12744 tablet_bootstrap.cc:654] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.847085 6654 catalog_manager.cc:5484] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:39.851189 9665 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:39.841475 12745 tablet_bootstrap.cc:492] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:39.854281 12745 tablet_bootstrap.cc:654] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.859865 12642 raft_consensus.cc:1050] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
I20240503 14:13:39.860785 12642 consensus_queue.cc:237] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:39.863976 12749 ts_tablet_manager.cc:902] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:39.864907 12716 tablet_copy_client.cc:308] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:39.873126 12745 tablet_bootstrap.cc:492] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:39.873239 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet b97ddea52b624c668cb0b28ab0f38196 from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:48587)
I20240503 14:13:39.873422 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet b97ddea52b624c668cb0b28ab0f38196 from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:48587: session id = e1bf5fb7402c46829974e2f74cd88708-b97ddea52b624c668cb0b28ab0f38196
I20240503 14:13:39.876722 12745 ts_tablet_manager.cc:1367] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240503 14:13:39.876869 9950 tablet_copy_source_session.cc:205] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:39.863886 6651 catalog_manager.cc:5484] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:39.878176 12745 raft_consensus.cc:348] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.883891 12745 raft_consensus.cc:374] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.884238 12745 raft_consensus.cc:729] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.892635 12741 tablet_bootstrap.cc:492] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:39.892794 9930 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.894429 12741 tablet_bootstrap.cc:654] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:39.901499 12716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b97ddea52b624c668cb0b28ab0f38196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.905911 7190 raft_consensus.cc:1261] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.906646 12652 tablet_bootstrap.cc:492] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:39.906630 12709 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.906869 12652 ts_tablet_manager.cc:1367] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.121s user 0.007s sys 0.000s
I20240503 14:13:39.907891 12652 raft_consensus.cc:348] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.908278 12652 raft_consensus.cc:374] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.908433 12652 raft_consensus.cc:729] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.908792 12652 consensus_queue.cc:260] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:39.912550 9929 raft_consensus.cc:1261] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.912564 12713 tablet_copy_client.cc:308] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:39.915211 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 58548735201648fa805fdf6af390c088 from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:46691)
I20240503 14:13:39.915381 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 58548735201648fa805fdf6af390c088 from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:46691: session id = 09723f236d8044538d920a37b5d9aa8c-58548735201648fa805fdf6af390c088
I20240503 14:13:39.918287 12714 ts_tablet_manager.cc:1888] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:39.918433 12714 log.cc:1196] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/4b09a64c5814437184354829e46072a1
I20240503 14:13:39.918932 12714 ts_tablet_manager.cc:1909] T 4b09a64c5814437184354829e46072a1 P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:39.919267 9819 tablet_copy_source_session.cc:205] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:39.921715 9797 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:39.922278 12254 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:39.924103 9798 consensus_queue.cc:237] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:39.929764 7056 raft_consensus.cc:1261] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:39.930948 12713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58548735201648fa805fdf6af390c088. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:39.931006 12634 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240503 14:13:39.934478 9929 raft_consensus.cc:1261] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:39.935413 12717 consensus_queue.cc:1035] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:39.936944 7056 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.939496 9929 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.941781 6931 raft_consensus.cc:1261] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:39.943439 12634 raft_consensus.cc:2932] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.946501 6654 catalog_manager.cc:5484] T 13f41f236987487b8bca287c7ec1fdd3 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:39.909752 12652 ts_tablet_manager.cc:1398] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:39.952421 12744 tablet_bootstrap.cc:492] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:39.953001 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 13f41f236987487b8bca287c7ec1fdd3 with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:39.953562 12736 tablet_replica.cc:330] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:39.953958 12736 raft_consensus.cc:2229] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:39.954351 12736 raft_consensus.cc:2258] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:39.955998 7188 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:39.957652 9664 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:39.958622 12254 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:39.959360 12744 ts_tablet_manager.cc:1367] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.124s user 0.012s sys 0.000s
I20240503 14:13:39.959692 12015 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.959761 12654 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:39.960376 12015 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:39.967058 6929 consensus_queue.cc:237] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:39.967947 12736 ts_tablet_manager.cc:1875] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:39.973583 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 13f41f236987487b8bca287c7ec1fdd3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:39.975064 12744 raft_consensus.cc:348] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:39.975337 12744 raft_consensus.cc:374] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:39.975435 12744 raft_consensus.cc:729] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:39.978631 7188 raft_consensus.cc:1261] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:39.980015 12659 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:39.984422 9665 raft_consensus.cc:1261] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:39.984153 12659 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:39.989207 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b6d600a8b0134a018f8c9e5a1d9b79c5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:39.989584 12659 consensus_queue.cc:1035] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:39.991467 7188 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:39.993755 12714 tablet_replica.cc:330] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:39.995229 9665 raft_consensus.cc:2932] T b6d600a8b0134a018f8c9e5a1d9b79c5 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.005690 6653 catalog_manager.cc:5484] T b6d600a8b0134a018f8c9e5a1d9b79c5 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.006966 12749 tablet_copy_client.cc:308] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:34101
I20240503 14:13:39.975697 12744 consensus_queue.cc:260] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.011441 6820 tablet_copy_service.cc:140] P f1a669e28f014fe287932acf56bebc13: Received BeginTabletCopySession request for tablet 1949c165716e42cf89d542e6b9b7ae71 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:33179)
I20240503 14:13:40.011751 6820 tablet_copy_service.cc:161] P f1a669e28f014fe287932acf56bebc13: Beginning new tablet copy session on tablet 1949c165716e42cf89d542e6b9b7ae71 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:33179: session id = 7186ed8510424fd597173eaf10c57b12-1949c165716e42cf89d542e6b9b7ae71
I20240503 14:13:40.014199 12741 tablet_bootstrap.cc:492] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:40.014569 12741 ts_tablet_manager.cc:1367] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.122s user 0.000s sys 0.005s
I20240503 14:13:40.014513 6930 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.014945 6820 tablet_copy_source_session.cc:205] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:40.008585 12015 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:39.993345 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 4b09a64c5814437184354829e46072a1 (table rebalancer_dynamic_table_new_name [id=9ffc1e1686fb45f487675af12101b8be]) successfully deleted
I20240503 14:13:40.016865 12741 raft_consensus.cc:348] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.017125 12741 raft_consensus.cc:374] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.017218 12741 raft_consensus.cc:729] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.017510 12741 consensus_queue.cc:260] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.018339 12741 ts_tablet_manager.cc:1398] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:40.012651 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet b6d600a8b0134a018f8c9e5a1d9b79c5 with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:40.021329 12716 tablet_copy_client.cc:783] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:40.021776 12744 ts_tablet_manager.cc:1398] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20240503 14:13:40.022521 12749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1949c165716e42cf89d542e6b9b7ae71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.023018 12713 tablet_copy_client.cc:783] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:40.031760 12716 tablet_copy_client.cc:647] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:40.036737 12661 ts_tablet_manager.cc:902] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:40.038729 12714 raft_consensus.cc:2229] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:40.041923 12731 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.047480 12714 raft_consensus.cc:2258] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.047842 12749 tablet_copy_client.cc:783] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:40.048316 12749 tablet_copy_client.cc:647] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:40.048530 12714 ts_tablet_manager.cc:1875] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:40.050395 7188 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.050946 6653 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:40.052258 12749 tablet_copy_client.cc:523] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:39.989686 6929 consensus_queue.cc:237] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:40.056943 12749 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:40.060941 9797 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.065265 9929 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.067631 12713 tablet_copy_client.cc:647] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:40.069418 12731 raft_consensus.cc:482] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.069681 12731 raft_consensus.cc:504] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:40.070250 9664 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.070953 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:40.072307 12731 leader_election.cc:288] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:40.073019 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:40.073752 6867 leader_election.cc:302] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12; no voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:40.074337 12728 raft_consensus.cc:2726] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:40.073817 7190 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.081372 12713 tablet_copy_client.cc:523] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:40.082544 12661 tablet_copy_client.cc:308] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.5:36609
I20240503 14:13:40.083420 9553 tablet_copy_service.cc:140] P 5b7e36bf354f480c90859cb8867b59cf: Received BeginTabletCopySession request for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:33205)
I20240503 14:13:40.083734 9553 tablet_copy_service.cc:161] P 5b7e36bf354f480c90859cb8867b59cf: Beginning new tablet copy session on tablet d9641f7f5a3e4debb5aabc6b8a7497f6 from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:33205: session id = f1a669e28f014fe287932acf56bebc13-d9641f7f5a3e4debb5aabc6b8a7497f6
I20240503 14:13:40.088728 9553 tablet_copy_source_session.cc:205] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:40.090817 6651 catalog_manager.cc:5484] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.091692 12661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9641f7f5a3e4debb5aabc6b8a7497f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.095321 12716 tablet_copy_client.cc:523] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:39.884781 12745 consensus_queue.cc:260] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.096259 12745 ts_tablet_manager.cc:1398] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.219s user 0.002s sys 0.000s
I20240503 14:13:40.099371 12716 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:40.101928 12714 ts_tablet_manager.cc:1888] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:40.102123 12714 log.cc:1196] T 89ff0991a4e245609405ff945388417b P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/89ff0991a4e245609405ff945388417b
I20240503 14:13:40.107041 12779 tablet_replica.cc:330] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:40.107455 12779 raft_consensus.cc:2229] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:40.110095 7056 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:13:40.110308 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3e2d7664e6c45d69f4f8b20dcf0e4d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:40.110869 12728 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:40.112571 12736 ts_tablet_manager.cc:1888] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:40.112702 12736 log.cc:1196] T f6f8dfcb870a4461b03415db9c33452e P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/f6f8dfcb870a4461b03415db9c33452e
I20240503 14:13:40.113478 12736 tablet_replica.cc:330] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:40.113693 12736 raft_consensus.cc:2229] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:40.113844 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet f6f8dfcb870a4461b03415db9c33452e (table rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]) successfully deleted
I20240503 14:13:40.113960 12736 raft_consensus.cc:2258] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.117168 12713 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:40.120397 9530 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:40.121121 12731 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:40.122093 9531 consensus_queue.cc:237] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:40.135748 12736 ts_tablet_manager.cc:1875] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:13:40.140079 9467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 57751.8us
I20240503 14:13:40.141680 6930 raft_consensus.cc:1261] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:40.141659 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 89ff0991a4e245609405ff945388417b (table rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]) successfully deleted
I20240503 14:13:40.142197 12661 tablet_copy_client.cc:783] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:40.143677 12661 tablet_copy_client.cc:647] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
W20240503 14:13:40.144918 7125 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletCopyService.FetchData -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 49977.8us
I20240503 14:13:40.148941 12015 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:40.149979 12749 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.150535 12749 tablet_bootstrap.cc:492] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:40.150856 12749 ts_tablet_manager.cc:1367] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.094s user 0.017s sys 0.013s
I20240503 14:13:40.151683 12749 raft_consensus.cc:348] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:40.155176 9531 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:40.156402 12015 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.160305 12713 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.160789 12713 tablet_bootstrap.cc:492] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:40.161052 12713 ts_tablet_manager.cc:1367] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20240503 14:13:40.161845 12713 raft_consensus.cc:348] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:40.162097 12713 raft_consensus.cc:729] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:40.162146 12779 raft_consensus.cc:2258] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.162420 12736 ts_tablet_manager.cc:1888] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:40.162577 12736 log.cc:1196] T 13f41f236987487b8bca287c7ec1fdd3 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/13f41f236987487b8bca287c7ec1fdd3
I20240503 14:13:40.163343 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 13f41f236987487b8bca287c7ec1fdd3 (table rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]) successfully deleted
I20240503 14:13:40.163266 12713 consensus_queue.cc:260] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:40.164017 12713 ts_tablet_manager.cc:1398] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:40.166504 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-58548735201648fa805fdf6af390c088 received from {username='slave'} at 127.0.186.3:46691
I20240503 14:13:40.166677 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-58548735201648fa805fdf6af390c088 on tablet 58548735201648fa805fdf6af390c088 with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:40.165800 6657 catalog_manager.cc:5484] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 6 to 7, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:40.168591 12779 ts_tablet_manager.cc:1875] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:40.172026 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4b86d126b92400f9b14a9e4fb527fc9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:40.061093 12712 ts_tablet_manager.cc:1888] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:40.174639 12712 log.cc:1196] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/4b09a64c5814437184354829e46072a1
I20240503 14:13:40.153450 12730 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:40.161180 6930 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.180143 12749 raft_consensus.cc:729] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:40.180455 12749 consensus_queue.cc:260] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:40.181277 12749 ts_tablet_manager.cc:1398] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20240503 14:13:40.182994 6820 tablet_copy_service.cc:342] P f1a669e28f014fe287932acf56bebc13: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-1949c165716e42cf89d542e6b9b7ae71 received from {username='slave'} at 127.0.186.2:33179
I20240503 14:13:40.183115 6820 tablet_copy_service.cc:433] P f1a669e28f014fe287932acf56bebc13: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-1949c165716e42cf89d542e6b9b7ae71 on tablet 1949c165716e42cf89d542e6b9b7ae71 with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:40.174333 7057 raft_consensus.cc:1203] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:40.187942 12712 ts_tablet_manager.cc:1909] T 4b09a64c5814437184354829e46072a1 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:40.188977 6652 catalog_manager.cc:5484] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:40.194619 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 4b09a64c5814437184354829e46072a1 (table rebalancer_dynamic_table_new_name [id=9ffc1e1686fb45f487675af12101b8be]) successfully deleted
W20240503 14:13:40.195117 9468 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 69928.1us
I20240503 14:13:40.195936 7188 raft_consensus.cc:1261] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:40.197618 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet f4b86d126b92400f9b14a9e4fb527fc9 with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:40.198014 12714 tablet_replica.cc:330] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:40.198243 12714 raft_consensus.cc:2229] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:40.198328 12708 consensus_queue.cc:1035] T f4b86d126b92400f9b14a9e4fb527fc9 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:40.198498 12714 raft_consensus.cc:2258] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.199291 12661 tablet_copy_client.cc:523] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:40.200394 12714 ts_tablet_manager.cc:1875] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:40.201578 7188 raft_consensus.cc:2932] T f4b86d126b92400f9b14a9e4fb527fc9 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:40.205267 12661 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:40.211683 12714 ts_tablet_manager.cc:1888] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:40.212213 7056 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:40.215175 12716 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.215579 12714 log.cc:1196] T f4b86d126b92400f9b14a9e4fb527fc9 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/f4b86d126b92400f9b14a9e4fb527fc9
I20240503 14:13:40.216975 12716 tablet_bootstrap.cc:492] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:40.217339 12716 ts_tablet_manager.cc:1367] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.118s user 0.006s sys 0.006s
I20240503 14:13:40.224079 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet f4b86d126b92400f9b14a9e4fb527fc9 (table rebalancer_extra_table_26 [id=8977c9cf5e38426b958094ce1e7f79e9]) successfully deleted
I20240503 14:13:40.224287 12716 raft_consensus.cc:348] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:40.224539 12716 raft_consensus.cc:729] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:40.224750 12716 consensus_queue.cc:260] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:40.230613 12716 ts_tablet_manager.cc:1398] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 14:13:40.235496 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-b97ddea52b624c668cb0b28ab0f38196 received from {username='slave'} at 127.0.186.4:48587
I20240503 14:13:40.235684 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-b97ddea52b624c668cb0b28ab0f38196 on tablet b97ddea52b624c668cb0b28ab0f38196 with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:40.238001 12661 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:40.244712 12249 raft_consensus.cc:482] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.244936 12661 tablet_bootstrap.cc:492] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:40.244935 12249 raft_consensus.cc:504] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.245783 12249 leader_election.cc:288] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:40.246846 12661 ts_tablet_manager.cc:1367] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.042s user 0.011s sys 0.000s
I20240503 14:13:40.247211 10371 raft_consensus.cc:482] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.247421 10371 raft_consensus.cc:504] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.248131 10371 leader_election.cc:288] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:40.248936 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33058d2747b743bd97367234216a4694" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:40.248951 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33058d2747b743bd97367234216a4694" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:40.249147 9665 raft_consensus.cc:2454] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:40.249199 9530 raft_consensus.cc:2454] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:40.249706 6997 leader_election.cc:302] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:40.250262 10371 raft_consensus.cc:2781] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:40.250375 10371 raft_consensus.cc:482] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:40.250499 10371 raft_consensus.cc:3037] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.251892 11977 raft_consensus.cc:1050] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:40.252813 11977 consensus_queue.cc:237] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:40.253381 10371 raft_consensus.cc:504] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.255537 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33058d2747b743bd97367234216a4694" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:40.255815 9665 raft_consensus.cc:3037] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.259416 9665 raft_consensus.cc:2454] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:40.259827 10371 leader_election.cc:288] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:40.260553 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33058d2747b743bd97367234216a4694" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:40.260833 9530 raft_consensus.cc:3037] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.264000 9530 raft_consensus.cc:2454] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:40.264667 6997 leader_election.cc:302] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:40.265720 10371 raft_consensus.cc:2781] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:40.265884 6798 raft_consensus.cc:1261] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.266418 10371 raft_consensus.cc:686] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:40.266737 12313 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.266764 10371 consensus_queue.cc:237] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.267323 12661 raft_consensus.cc:348] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:40.267606 12661 raft_consensus.cc:729] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:40.267803 12661 consensus_queue.cc:260] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:40.272650 9530 raft_consensus.cc:1261] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.272727 7188 raft_consensus.cc:1261] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.273378 12292 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.275707 11977 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.278041 12661 ts_tablet_manager.cc:1398] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240503 14:13:40.283782 9553 tablet_copy_service.cc:342] P 5b7e36bf354f480c90859cb8867b59cf: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-d9641f7f5a3e4debb5aabc6b8a7497f6 received from {username='slave'} at 127.0.186.1:33205
I20240503 14:13:40.283953 9553 tablet_copy_service.cc:433] P 5b7e36bf354f480c90859cb8867b59cf: ending tablet copy session f1a669e28f014fe287932acf56bebc13-d9641f7f5a3e4debb5aabc6b8a7497f6 on tablet d9641f7f5a3e4debb5aabc6b8a7497f6 with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:40.288578 12292 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:40.295063 9531 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:40.297489 6800 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:40.302170 6654 catalog_manager.cc:5484] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:40.313978 12779 ts_tablet_manager.cc:1888] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:40.314225 12779 log.cc:1196] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/b6d600a8b0134a018f8c9e5a1d9b79c5
I20240503 14:13:40.315199 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet b6d600a8b0134a018f8c9e5a1d9b79c5 (table rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]) successfully deleted
I20240503 14:13:40.321055 12731 raft_consensus.cc:482] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.321265 12731 raft_consensus.cc:504] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.322145 12731 leader_election.cc:288] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:40.322810 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3b35a8172d478ba2b4717569ee993e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:40.323072 6800 raft_consensus.cc:2379] T 8d3b35a8172d478ba2b4717569ee993e P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7186ed8510424fd597173eaf10c57b12 in current term 1: Already voted for candidate 5b7e36bf354f480c90859cb8867b59cf in this term.
I20240503 14:13:40.323563 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3b35a8172d478ba2b4717569ee993e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:40.324187 6930 raft_consensus.cc:3037] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.324678 6867 leader_election.cc:302] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12; no voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:40.330621 12731 raft_consensus.cc:2726] T 8d3b35a8172d478ba2b4717569ee993e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:40.330981 12708 consensus_queue.cc:1035] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.333976 7188 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:40.337751 9929 consensus_queue.cc:237] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:40.339995 6930 raft_consensus.cc:3037] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.340026 6800 raft_consensus.cc:1261] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:40.340705 12313 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:40.343554 12292 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:40.345894 6655 catalog_manager.cc:5484] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:40.346339 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33058d2747b743bd97367234216a4694" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:40.347306 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33058d2747b743bd97367234216a4694" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:40.347570 9530 raft_consensus.cc:2379] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f03dc37fe3234033bc665bed4f1b51f5 in current term 1: Already voted for candidate 09723f236d8044538d920a37b5d9aa8c in this term.
I20240503 14:13:40.351240 6931 raft_consensus.cc:1203] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:40.357010 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 005dec40a9054f7fa95ea8722e754e24 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:40.357445 12460 raft_consensus.cc:1050] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:40.357416 6800 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:40.358304 12460 consensus_queue.cc:237] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:40.361480 9797 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:40.361728 6931 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:40.362633 12805 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:40.363886 12806 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:40.374553 6652 catalog_manager.cc:5484] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:40.376335 9665 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:40.378878 9601 leader_election.cc:302] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5; no voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:40.379240 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 005dec40a9054f7fa95ea8722e754e24 with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:40.379680 12779 tablet_replica.cc:330] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:40.379926 12779 raft_consensus.cc:2229] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:40.380198 12779 raft_consensus.cc:2258] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.380175 12806 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:40.380522 12249 raft_consensus.cc:2726] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:40.381244 12779 ts_tablet_manager.cc:1875] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:40.389529 12779 ts_tablet_manager.cc:1888] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:40.389760 12779 log.cc:1196] T 005dec40a9054f7fa95ea8722e754e24 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/005dec40a9054f7fa95ea8722e754e24
I20240503 14:13:40.390775 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 005dec40a9054f7fa95ea8722e754e24 (table rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]) successfully deleted
I20240503 14:13:40.392026 12708 consensus_queue.cc:1035] T 8d3b35a8172d478ba2b4717569ee993e P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.411208 7190 raft_consensus.cc:1261] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:40.413661 12313 consensus_queue.cc:1035] T 005dec40a9054f7fa95ea8722e754e24 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:40.416111 7188 raft_consensus.cc:2932] T 005dec40a9054f7fa95ea8722e754e24 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:40.419497 12805 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.425489 6930 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.426102 9665 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.426654 9795 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.435012 6654 catalog_manager.cc:5484] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 7 to 8, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.443478 9664 consensus_queue.cc:237] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:40.451711 7056 consensus_queue.cc:237] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:40.457453 6800 raft_consensus.cc:1203] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:40.462983 9797 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:40.464779 9665 raft_consensus.cc:1261] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:40.465257 12806 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 14:13:40.466113 9600 consensus_peers.cc:485] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 603002fdf6b84d3ea63c65636eb42cb1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:40.466024 12460 consensus_queue.cc:1035] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:40.467278 9929 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:40.467635 9530 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:40.468173 12653 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:40.468731 12653 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:40.469249 12460 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.473493 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a44ed92d65114bfda2e0f38054b17ada with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:40.476815 6653 catalog_manager.cc:5484] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 8 to 9, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:40.478888 9664 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.485502 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet a44ed92d65114bfda2e0f38054b17ada with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:40.494400 6652 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:40.499694 12249 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:40.504889 9531 raft_consensus.cc:1261] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:40.505312 9664 raft_consensus.cc:1261] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:40.505980 12813 consensus_queue.cc:1035] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.506888 12813 consensus_queue.cc:1035] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:40.507457 9930 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:40.520633 6652 catalog_manager.cc:5484] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:40.520923 6653 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:40.529285 9798 raft_consensus.cc:2932] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:40.545178 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 122296b00eda44949b9064d0afdd6a16 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.545739 9645 tablet_service.cc:1467] Processing CreateTablet for tablet 122296b00eda44949b9064d0afdd6a16 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.545918 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122296b00eda44949b9064d0afdd6a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.546355 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122296b00eda44949b9064d0afdd6a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.549266 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 00ecf4e868774482b4600566943fd224 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.549393 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 122296b00eda44949b9064d0afdd6a16 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.549793 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ecf4e868774482b4600566943fd224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.550089 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122296b00eda44949b9064d0afdd6a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.551108 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 00ecf4e868774482b4600566943fd224 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.551560 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ecf4e868774482b4600566943fd224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.554174 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 00ecf4e868774482b4600566943fd224 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.554600 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ecf4e868774482b4600566943fd224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.556061 9910 tablet_service.cc:1467] Processing CreateTablet for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.556502 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.563045 9643 tablet_service.cc:1467] Processing CreateTablet for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.563675 9643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.567056 9777 tablet_service.cc:1467] Processing CreateTablet for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=8a855355504b4589a2a21a879709066b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:40.567684 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:40.569890 12823 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:40.571714 12823 tablet_bootstrap.cc:654] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.574728 12823 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:40.575485 12823 ts_tablet_manager.cc:1367] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240503 14:13:40.600847 6652 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 33058d2747b743bd97367234216a4694 (table rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364]). Expected version 1 got 0
I20240503 14:13:40.609529 7188 raft_consensus.cc:1203] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:40.620407 12830 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:40.623047 12830 tablet_bootstrap.cc:654] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.630306 12824 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:40.632869 12824 tablet_bootstrap.cc:654] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.634407 12823 raft_consensus.cc:348] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.634932 12823 raft_consensus.cc:374] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.643915 12826 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:40.644526 12826 tablet.cc:1620] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:40.650871 12833 tablet_bootstrap.cc:492] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:40.655047 12823 raft_consensus.cc:729] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.656991 12833 tablet_bootstrap.cc:654] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.656162 12823 consensus_queue.cc:260] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.658602 12823 ts_tablet_manager.cc:1398] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.035s user 0.000s sys 0.002s
I20240503 14:13:40.660190 12823 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:40.661176 12827 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:40.662770 12823 tablet_bootstrap.cc:654] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.667388 12817 tablet_replica.cc:330] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:40.667771 12817 raft_consensus.cc:2229] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:40.668248 12817 raft_consensus.cc:2258] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:40.668263 6654 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 33058d2747b743bd97367234216a4694 (table rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364]). Expected version 1 got 0
I20240503 14:13:40.669138 12828 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:40.671367 12828 tablet.cc:1620] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:40.674908 6654 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 33058d2747b743bd97367234216a4694 (table rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364]). Expected version 1 got 0
I20240503 14:13:40.678192 12823 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:40.678471 12823 ts_tablet_manager.cc:1367] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.006s
I20240503 14:13:40.679003 12827 tablet_bootstrap.cc:654] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.681070 12817 ts_tablet_manager.cc:1875] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:40.681767 12823 raft_consensus.cc:348] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:40.682353 12823 raft_consensus.cc:374] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.682718 12823 raft_consensus.cc:729] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.683089 12823 consensus_queue.cc:260] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:40.683894 12823 ts_tablet_manager.cc:1398] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:40.692550 9530 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:40.699523 12823 tablet_bootstrap.cc:492] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:40.700546 6652 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364] alter complete (version 1)
I20240503 14:13:40.706162 12717 raft_consensus.cc:1050] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:40.706611 12823 tablet_bootstrap.cc:654] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.707870 12717 consensus_queue.cc:237] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:40.712525 12822 ts_tablet_manager.cc:902] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:40.714571 12841 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:40.715328 12841 tablet.cc:1620] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:40.724460 6654 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet 33058d2747b743bd97367234216a4694 (table rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364]). Expected version 1 got 0
I20240503 14:13:40.725345 12823 tablet_bootstrap.cc:492] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:40.725637 12823 ts_tablet_manager.cc:1367] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240503 14:13:40.726572 12823 raft_consensus.cc:348] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.726923 12823 raft_consensus.cc:374] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.727111 12823 raft_consensus.cc:729] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.727422 12823 consensus_queue.cc:260] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.728129 12823 ts_tablet_manager.cc:1398] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:40.743914 9930 consensus_queue.cc:237] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:40.747403 12830 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:40.747655 12830 ts_tablet_manager.cc:1367] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.127s user 0.025s sys 0.025s
I20240503 14:13:40.748373 12830 raft_consensus.cc:348] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.748678 12830 raft_consensus.cc:374] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.748775 12830 raft_consensus.cc:729] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.749038 12830 consensus_queue.cc:260] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.750841 12830 ts_tablet_manager.cc:1398] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:40.752600 12827 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:40.753542 12833 tablet_bootstrap.cc:492] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:40.756830 7056 raft_consensus.cc:1261] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.757184 12827 ts_tablet_manager.cc:1367] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.097s user 0.000s sys 0.004s
I20240503 14:13:40.757898 6800 raft_consensus.cc:1261] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:40.758126 12633 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.758596 12313 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:40.760442 9797 raft_consensus.cc:1261] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:40.765304 12292 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:40.769994 12833 ts_tablet_manager.cc:1367] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.119s user 0.004s sys 0.000s
I20240503 14:13:40.771214 12833 raft_consensus.cc:348] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.772629 12827 raft_consensus.cc:348] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.773017 12833 raft_consensus.cc:374] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.773188 12827 raft_consensus.cc:374] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.773326 12833 raft_consensus.cc:729] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.773527 12827 raft_consensus.cc:729] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.773871 12833 consensus_queue.cc:260] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.774451 12827 consensus_queue.cc:260] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.775139 12833 ts_tablet_manager.cc:1398] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:40.777693 12827 ts_tablet_manager.cc:1398] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20240503 14:13:40.783063 9865 consensus_peers.cc:485] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8811b896c0504775b03baf4925026adf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:40.786824 9531 raft_consensus.cc:1261] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.785691 9665 raft_consensus.cc:1261] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.792258 12717 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.794080 12827 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:40.795377 12633 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.799655 12827 tablet_bootstrap.cc:654] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.811149 12313 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:40.815438 9798 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:40.824534 12827 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:40.824875 12827 ts_tablet_manager.cc:1367] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20240503 14:13:40.825873 12827 raft_consensus.cc:348] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:40.826201 12827 raft_consensus.cc:374] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.826390 12827 raft_consensus.cc:729] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.826751 12827 consensus_queue.cc:260] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:40.827229 12824 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:40.827443 12824 ts_tablet_manager.cc:1367] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.197s user 0.042s sys 0.047s
I20240503 14:13:40.828640 12824 raft_consensus.cc:348] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:40.828796 12827 ts_tablet_manager.cc:1398] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:40.828958 12824 raft_consensus.cc:374] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.829067 12824 raft_consensus.cc:729] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.829252 12827 tablet_bootstrap.cc:492] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:40.829310 12824 consensus_queue.cc:260] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:40.830353 12634 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:40.834205 12824 ts_tablet_manager.cc:1398] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:40.835078 12827 tablet_bootstrap.cc:654] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:40.837625 6653 catalog_manager.cc:5484] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:40.847069 9665 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:40.858230 12817 ts_tablet_manager.cc:1888] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:13:40.859943 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:40.860456 12817 log.cc:1196] T a44ed92d65114bfda2e0f38054b17ada P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/a44ed92d65114bfda2e0f38054b17ada
I20240503 14:13:40.866768 6653 catalog_manager.cc:5484] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:40.869745 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet a44ed92d65114bfda2e0f38054b17ada (table rebalance_test_table_1 [id=a6850d2548a9479f8a2715173dd85dde]) successfully deleted
I20240503 14:13:40.870136 12826 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.870600 12826 tablet.cc:1620] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:40.870976 12828 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.871443 12828 tablet.cc:1620] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:40.871748 12717 raft_consensus.cc:482] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.871973 12717 raft_consensus.cc:504] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:40.872660 12841 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:40.873133 12841 tablet.cc:1620] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:40.873728 12717 leader_election.cc:288] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:40.878777 12691 raft_consensus.cc:1050] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:40.879565 12691 consensus_queue.cc:237] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:40.884322 9798 raft_consensus.cc:1261] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.885164 12327 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.887355 6930 raft_consensus.cc:1261] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.888172 12327 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.889361 9529 raft_consensus.cc:1261] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:40.892715 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbbd9f57ea9f457ab0ad4e59bc0f34bf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:40.892869 12327 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:40.892971 9662 raft_consensus.cc:2454] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:40.900233 12327 raft_consensus.cc:482] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.900421 12327 raft_consensus.cc:504] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.901227 12327 leader_election.cc:288] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:40.902294 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ecf4e868774482b4600566943fd224" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:40.902652 9665 raft_consensus.cc:2454] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:40.903368 6734 leader_election.cc:302] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:40.904292 12327 raft_consensus.cc:2781] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:40.904392 12327 raft_consensus.cc:482] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:40.904469 12327 raft_consensus.cc:3037] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.911767 7056 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:40.919728 12327 raft_consensus.cc:504] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.927541 6657 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=ea71c50d48854835a7d59b711d8d1364] alter complete (version 2)
I20240503 14:13:40.928967 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ecf4e868774482b4600566943fd224" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:40.929386 9665 raft_consensus.cc:3037] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.933459 12327 leader_election.cc:288] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:40.936785 9665 raft_consensus.cc:2454] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:40.938253 6734 leader_election.cc:302] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:40.938936 12327 raft_consensus.cc:2781] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:40.939893 12327 raft_consensus.cc:686] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:40.940292 12327 consensus_queue.cc:237] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.954988 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbbd9f57ea9f457ab0ad4e59bc0f34bf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:40.955746 9930 raft_consensus.cc:2454] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
W20240503 14:13:40.956523 9863 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 84273.7us
I20240503 14:13:40.960847 12774 raft_consensus.cc:482] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.961033 12774 raft_consensus.cc:504] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.962002 12774 leader_election.cc:288] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:40.963817 12827 tablet_bootstrap.cc:492] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:40.963989 12827 ts_tablet_manager.cc:1367] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.135s user 0.033s sys 0.042s
I20240503 14:13:40.964759 12827 raft_consensus.cc:348] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.964998 12827 raft_consensus.cc:374] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:40.965081 12827 raft_consensus.cc:729] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:40.963284 6800 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:40.965286 12827 consensus_queue.cc:260] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.965868 12827 ts_tablet_manager.cc:1398] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:40.971935 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122296b00eda44949b9064d0afdd6a16" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:40.972393 9665 raft_consensus.cc:2454] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:40.972926 7125 leader_election.cc:302] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:40.973299 12774 raft_consensus.cc:2781] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:40.973379 12774 raft_consensus.cc:482] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:40.973436 12774 raft_consensus.cc:3037] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.976842 12774 raft_consensus.cc:504] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:40.978261 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122296b00eda44949b9064d0afdd6a16" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:40.978544 9663 raft_consensus.cc:3037] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.979110 12691 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:40.980237 9529 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:40.983798 9663 raft_consensus.cc:2454] T 122296b00eda44949b9064d0afdd6a16 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:40.984118 6929 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:40.989713 7125 leader_election.cc:302] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:40.989707 12633 raft_consensus.cc:482] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:40.990018 12411 raft_consensus.cc:2781] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:40.990119 12633 raft_consensus.cc:504] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:40.990849 12633 leader_election.cc:288] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:40.991039 9733 leader_election.cc:302] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
W20240503 14:13:40.991258 9733 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 97681.4us
I20240503 14:13:40.994655 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ecf4e868774482b4600566943fd224" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:40.995676 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122296b00eda44949b9064d0afdd6a16" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:40.995915 9799 raft_consensus.cc:3037] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:40.999305 12774 leader_election.cc:288] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:41.000679 12822 tablet_copy_client.cc:308] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:41.001422 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:60021)
I20240503 14:13:41.001569 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:60021: session id = f03dc37fe3234033bc665bed4f1b51f5-b3e2d7664e6c45d69f4f8b20dcf0e4d0
I20240503 14:13:41.000950 9795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "122296b00eda44949b9064d0afdd6a16" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:41.004248 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ecf4e868774482b4600566943fd224" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:41.004385 6951 tablet_copy_source_session.cc:205] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:41.004532 9663 raft_consensus.cc:2379] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3efdbd3506274dba846ae40404f0572c in current term 1: Already voted for candidate f1a669e28f014fe287932acf56bebc13 in this term.
I20240503 14:13:41.004765 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ecf4e868774482b4600566943fd224" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:41.005156 9799 raft_consensus.cc:2454] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:41.006175 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.006232 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ecf4e868774482b4600566943fd224" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:41.007381 12633 raft_consensus.cc:2781] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:41.007612 12633 raft_consensus.cc:482] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:41.008953 12633 raft_consensus.cc:3037] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.010442 7056 consensus_queue.cc:237] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:41.012830 9733 leader_election.cc:302] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:41.013296 9796 raft_consensus.cc:3037] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.013459 12822 tablet_copy_client.cc:783] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:41.017189 12633 raft_consensus.cc:504] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.019800 12633 leader_election.cc:288] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:41.020298 12822 tablet_copy_client.cc:647] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:41.022864 12822 tablet_copy_client.cc:523] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:41.021379 9796 raft_consensus.cc:2454] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:41.033746 12853 raft_consensus.cc:2726] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:41.035553 12822 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:41.036880 6657 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:41.039575 6655 catalog_manager.cc:5484] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:41.045173 12015 raft_consensus.cc:1050] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:41.046028 12015 consensus_queue.cc:237] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:41.049312 12826 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:41.050479 12828 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:41.056584 12841 tablet.cc:1722] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:41.061443 6653 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=ea71c50d48854835a7d59b711d8d1364] alter complete (version 3)
W20240503 14:13:41.069358 6995 consensus_peers.cc:485] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d3ec4f0833d4d489f774757312efb32. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:41.070684 9533 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:13:41.071692 12813 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:41.075208 6798 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:41.076778 12789 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:41.077209 9930 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:41.077869 12789 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:41.080904 6929 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:13:41.081672 12813 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:41.089818 12411 raft_consensus.cc:686] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:41.090667 12411 consensus_queue.cc:237] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:41.095162 12875 ts_tablet_manager.cc:902] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:41.095645 9796 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
W20240503 14:13:41.100945 9735 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 198404us
W20240503 14:13:41.101982 9733 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 96966.1us
I20240503 14:13:41.102665 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbbd9f57ea9f457ab0ad4e59bc0f34bf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:41.102922 9663 raft_consensus.cc:3037] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.104545 11977 raft_consensus.cc:1050] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
I20240503 14:13:41.104429 12813 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:41.105355 11977 consensus_queue.cc:237] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:41.105834 9663 raft_consensus.cc:2454] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:41.111843 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbbd9f57ea9f457ab0ad4e59bc0f34bf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:41.112083 9930 raft_consensus.cc:3037] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.115779 9930 raft_consensus.cc:2454] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:41.115607 6657 catalog_manager.cc:5484] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 7 to 8, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 2 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.116241 9529 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:41.116641 6654 catalog_manager.cc:5484] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.122642 9733 leader_election.cc:302] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:41.122987 12717 raft_consensus.cc:2781] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:41.123145 12717 raft_consensus.cc:686] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:41.123462 12717 consensus_queue.cc:237] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.129808 9798 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:41.140093 9529 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:41.143846 6930 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:41.145406 12313 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:41.146086 9662 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:41.147220 11977 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:41.147866 12015 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:41.150228 12015 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.153211 7188 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:41.154758 12856 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:41.149713 6929 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:41.157730 6799 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.157866 9930 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.173403 6651 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:41.173708 6651 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:41.177269 6655 catalog_manager.cc:5484] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.177578 6658 catalog_manager.cc:5484] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 5 to 6, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.180318 9798 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.189708 6651 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 33058d2747b743bd97367234216a4694
I20240503 14:13:41.190973 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 33058d2747b743bd97367234216a4694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:41.190790 6652 catalog_manager.cc:5484] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.191370 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 33058d2747b743bd97367234216a4694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:41.192279 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 33058d2747b743bd97367234216a4694 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:41 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:41.204010 12899 tablet_replica.cc:330] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:41.204367 12899 raft_consensus.cc:2229] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.204624 12899 raft_consensus.cc:2258] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.206758 9798 consensus_queue.cc:237] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:41.206995 12899 ts_tablet_manager.cc:1875] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.210111 6799 consensus_queue.cc:237] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:41.213416 9798 raft_consensus.cc:1261] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:41.214051 12870 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:41.216547 12858 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:41.222668 12856 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:41.226156 12899 ts_tablet_manager.cc:1888] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.228399 7056 raft_consensus.cc:1261] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:41.228730 9529 raft_consensus.cc:1261] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:41.228965 12899 log.cc:1196] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/33058d2747b743bd97367234216a4694
I20240503 14:13:41.229504 12899 ts_tablet_manager.cc:1909] T 33058d2747b743bd97367234216a4694 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:41.230785 12717 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:41.231374 12878 consensus_queue.cc:1035] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:41.231741 6650 catalog_manager.cc:5484] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.236958 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 33058d2747b743bd97367234216a4694 (table rebalancer_dynamic_table_new_name [id=ea71c50d48854835a7d59b711d8d1364]) successfully deleted
I20240503 14:13:41.212644 6930 raft_consensus.cc:1261] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:41.240846 12327 consensus_queue.cc:1035] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:41.242831 6930 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:41.227906 6931 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:41.250366 6652 catalog_manager.cc:5484] T 1949c165716e42cf89d542e6b9b7ae71 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.250686 12897 tablet_replica.cc:330] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:41.250986 12897 raft_consensus.cc:2229] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:41.251623 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1949c165716e42cf89d542e6b9b7ae71 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.251734 12897 raft_consensus.cc:2258] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.252566 12898 tablet_replica.cc:330] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:41.252900 12898 raft_consensus.cc:2229] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.253211 12898 raft_consensus.cc:2258] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.259299 12898 ts_tablet_manager.cc:1875] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.260372 12897 ts_tablet_manager.cc:1875] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:41.264148 12875 tablet_copy_client.cc:308] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.6:33971
I20240503 14:13:41.267426 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 1949c165716e42cf89d542e6b9b7ae71 with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:41.279347 12897 ts_tablet_manager.cc:1888] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.278935 9663 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:41.288322 9685 tablet_copy_service.cc:140] P f03dc37fe3234033bc665bed4f1b51f5: Received BeginTabletCopySession request for tablet 603002fdf6b84d3ea63c65636eb42cb1 from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:60235)
I20240503 14:13:41.288630 9685 tablet_copy_service.cc:161] P f03dc37fe3234033bc665bed4f1b51f5: Beginning new tablet copy session on tablet 603002fdf6b84d3ea63c65636eb42cb1 from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:60235: session id = e1bf5fb7402c46829974e2f74cd88708-603002fdf6b84d3ea63c65636eb42cb1
I20240503 14:13:41.290620 12898 ts_tablet_manager.cc:1888] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:41.290778 12898 log.cc:1196] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/33058d2747b743bd97367234216a4694
I20240503 14:13:41.291291 12898 ts_tablet_manager.cc:1909] T 33058d2747b743bd97367234216a4694 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:41.294773 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:41.295028 9685 tablet_copy_source_session.cc:205] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:41.295025 7188 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:41.296103 12897 log.cc:1196] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/33058d2747b743bd97367234216a4694
I20240503 14:13:41.297472 12897 ts_tablet_manager.cc:1909] T 33058d2747b743bd97367234216a4694 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:41.299842 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 33058d2747b743bd97367234216a4694 (table rebalancer_dynamic_table_new_name [id=ea71c50d48854835a7d59b711d8d1364]) successfully deleted
I20240503 14:13:41.301793 12898 tablet_replica.cc:330] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:41.301851 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 33058d2747b743bd97367234216a4694 (table rebalancer_dynamic_table_new_name [id=ea71c50d48854835a7d59b711d8d1364]) successfully deleted
I20240503 14:13:41.302510 12898 raft_consensus.cc:2229] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.303171 12898 raft_consensus.cc:2258] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.309253 12875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603002fdf6b84d3ea63c65636eb42cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.316525 12822 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:41.317938 12822 tablet_bootstrap.cc:492] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:41.318363 12822 ts_tablet_manager.cc:1367] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.287s user 0.049s sys 0.026s
I20240503 14:13:41.319363 12822 raft_consensus.cc:348] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:41.319870 12822 raft_consensus.cc:729] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:41.320232 12822 consensus_queue.cc:260] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:41.322696 12875 tablet_copy_client.cc:783] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:41.322782 12822 ts_tablet_manager.cc:1398] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:41.323065 12875 tablet_copy_client.cc:647] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:41.323753 12898 ts_tablet_manager.cc:1875] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.323954 9798 raft_consensus.cc:2932] T 1949c165716e42cf89d542e6b9b7ae71 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:41.324386 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-b3e2d7664e6c45d69f4f8b20dcf0e4d0 received from {username='slave'} at 127.0.186.6:60021
I20240503 14:13:41.324554 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-b3e2d7664e6c45d69f4f8b20dcf0e4d0 on tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:41.326346 12870 consensus_queue.cc:1035] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.343320 12327 consensus_queue.cc:1035] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.352450 12898 ts_tablet_manager.cc:1888] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:41.353018 12875 tablet_copy_client.cc:523] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:41.357004 12872 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:41.358712 12875 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:41.363857 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 58548735201648fa805fdf6af390c088 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.364660 12898 log.cc:1196] T 1949c165716e42cf89d542e6b9b7ae71 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/1949c165716e42cf89d542e6b9b7ae71
I20240503 14:13:41.366230 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 1949c165716e42cf89d542e6b9b7ae71 (table rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]) successfully deleted
I20240503 14:13:41.364889 12905 ts_tablet_manager.cc:902] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:41.370424 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 76b9a4e53aa24320bccf33a44ca4818b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12]), partition=
I20240503 14:13:41.370370 9533 consensus_queue.cc:237] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:41.370821 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b9a4e53aa24320bccf33a44ca4818b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.373108 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 76b9a4e53aa24320bccf33a44ca4818b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12]), partition=
I20240503 14:13:41.373458 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b9a4e53aa24320bccf33a44ca4818b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.373492 9930 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:41.373785 6799 raft_consensus.cc:1261] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:41.374140 12708 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:41.374789 12789 consensus_queue.cc:1035] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:41.376266 7056 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:41.377102 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 76b9a4e53aa24320bccf33a44ca4818b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12]), partition=
I20240503 14:13:41.377463 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b9a4e53aa24320bccf33a44ca4818b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.383527 12015 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.383764 6658 catalog_manager.cc:5484] T 58548735201648fa805fdf6af390c088 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:41.384286 9930 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.387470 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.388190 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 58548735201648fa805fdf6af390c088 with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:41.388571 12899 tablet_replica.cc:330] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:41.388834 12899 raft_consensus.cc:2229] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.389089 12899 raft_consensus.cc:2258] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.390144 12899 ts_tablet_manager.cc:1875] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.397228 12899 ts_tablet_manager.cc:1888] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:41.398303 12899 log.cc:1196] T 58548735201648fa805fdf6af390c088 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/58548735201648fa805fdf6af390c088
I20240503 14:13:41.399209 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 58548735201648fa805fdf6af390c088 (table rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]) successfully deleted
I20240503 14:13:41.400096 12918 tablet_bootstrap.cc:492] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:41.402383 12918 tablet_bootstrap.cc:654] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.405241 12914 tablet_bootstrap.cc:492] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:41.405081 6658 catalog_manager.cc:5484] T d9641f7f5a3e4debb5aabc6b8a7497f6 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 6 to 7, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.408190 12914 tablet_bootstrap.cc:654] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.412662 6799 raft_consensus.cc:2932] T d9641f7f5a3e4debb5aabc6b8a7497f6 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:41.415287 9665 raft_consensus.cc:1203] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:41.419413 12915 tablet_bootstrap.cc:492] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:41.421594 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet d9641f7f5a3e4debb5aabc6b8a7497f6 with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:41.424062 12915 tablet_bootstrap.cc:654] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.425874 12875 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:41.433545 12914 tablet_bootstrap.cc:492] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:41.433868 12914 ts_tablet_manager.cc:1367] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.000s
I20240503 14:13:41.433986 9533 raft_consensus.cc:2932] T 58548735201648fa805fdf6af390c088 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:41.434859 12914 raft_consensus.cc:348] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.436048 12914 raft_consensus.cc:374] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.436300 12914 raft_consensus.cc:729] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.436800 12914 consensus_queue.cc:260] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.437917 12914 ts_tablet_manager.cc:1398] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:41.446094 12875 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:41.446460 12875 ts_tablet_manager.cc:1367] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.004s
I20240503 14:13:41.447467 12875 raft_consensus.cc:348] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:41.447839 12875 raft_consensus.cc:729] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:41.448099 12875 consensus_queue.cc:260] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:41.452229 12905 tablet_copy_client.cc:308] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:41.453081 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet 8811b896c0504775b03baf4925026adf from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:49309)
I20240503 14:13:41.453284 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet 8811b896c0504775b03baf4925026adf from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:49309: session id = 5b7e36bf354f480c90859cb8867b59cf-8811b896c0504775b03baf4925026adf
I20240503 14:13:41.453907 12923 tablet_replica.cc:330] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:41.454126 12923 raft_consensus.cc:2229] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.454505 12923 raft_consensus.cc:2258] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.455407 12923 ts_tablet_manager.cc:1875] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.458055 9950 tablet_copy_source_session.cc:205] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:41.459793 12905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8811b896c0504775b03baf4925026adf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.460546 12915 tablet_bootstrap.cc:492] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:41.466512 12915 ts_tablet_manager.cc:1367] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240503 14:13:41.470289 12915 raft_consensus.cc:348] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
W20240503 14:13:41.472723 7125 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.2:45285, user_credentials={real_user=slave}} blocked reactor thread for 44592.9us
I20240503 14:13:41.475410 12822 ts_tablet_manager.cc:902] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:41.477277 12915 raft_consensus.cc:374] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.477540 12915 raft_consensus.cc:729] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.477952 12875 ts_tablet_manager.cc:1398] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20240503 14:13:41.478241 12915 consensus_queue.cc:260] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.481801 12915 ts_tablet_manager.cc:1398] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 14:13:41.485222 9663 consensus_queue.cc:237] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:41.489253 12923 ts_tablet_manager.cc:1888] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:13:41.492251 9685 tablet_copy_service.cc:342] P f03dc37fe3234033bc665bed4f1b51f5: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-603002fdf6b84d3ea63c65636eb42cb1 received from {username='slave'} at 127.0.186.4:60235
I20240503 14:13:41.492456 9685 tablet_copy_service.cc:433] P f03dc37fe3234033bc665bed4f1b51f5: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-603002fdf6b84d3ea63c65636eb42cb1 on tablet 603002fdf6b84d3ea63c65636eb42cb1 with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:41.496426 12923 log.cc:1196] T d9641f7f5a3e4debb5aabc6b8a7497f6 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/d9641f7f5a3e4debb5aabc6b8a7497f6
I20240503 14:13:41.504072 12918 tablet_bootstrap.cc:492] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:41.507071 12918 ts_tablet_manager.cc:1367] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.107s user 0.027s sys 0.018s
I20240503 14:13:41.507472 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet d9641f7f5a3e4debb5aabc6b8a7497f6 (table rebalancer_extra_table_8 [id=40b99cba8e664995b4918d85f4416551]) successfully deleted
I20240503 14:13:41.508134 12905 tablet_copy_client.cc:783] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:41.508077 12918 raft_consensus.cc:348] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.508278 12918 raft_consensus.cc:374] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.508421 12918 raft_consensus.cc:729] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.508718 12918 consensus_queue.cc:260] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.510939 12918 ts_tablet_manager.cc:1398] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:41.514334 12905 tablet_copy_client.cc:647] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:41.516649 12905 tablet_copy_client.cc:523] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:13:41.517328 9601 consensus_peers.cc:485] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 458ebc1295ee49999586eb699a29bbf8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:41.517503 6799 raft_consensus.cc:1261] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:41.518091 7188 raft_consensus.cc:1261] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:41.518117 12653 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:41.518679 12653 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:41.525619 12924 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:41.526489 7188 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:41.528963 12905 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:41.536041 6651 catalog_manager.cc:5484] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.538348 6799 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:41.552704 12872 consensus_queue.cc:1035] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.570034 12633 consensus_queue.cc:1035] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.588661 12822 tablet_copy_client.cc:308] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:41.590880 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 1d3ec4f0833d4d489f774757312efb32 from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:48891)
I20240503 14:13:41.591029 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 1d3ec4f0833d4d489f774757312efb32 from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:48891: session id = f03dc37fe3234033bc665bed4f1b51f5-1d3ec4f0833d4d489f774757312efb32
I20240503 14:13:41.593909 7081 tablet_copy_source_session.cc:205] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:41.594920 12905 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:41.595502 12905 tablet_bootstrap.cc:492] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:41.596926 12822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d3ec4f0833d4d489f774757312efb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.602109 12905 ts_tablet_manager.cc:1367] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.004s
I20240503 14:13:41.603030 12905 raft_consensus.cc:348] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:41.609891 12905 raft_consensus.cc:729] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:41.610117 12905 consensus_queue.cc:260] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:41.611603 12905 ts_tablet_manager.cc:1398] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:41.612591 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-8811b896c0504775b03baf4925026adf received from {username='slave'} at 127.0.186.5:49309
I20240503 14:13:41.612741 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-8811b896c0504775b03baf4925026adf on tablet 8811b896c0504775b03baf4925026adf with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:41.618726 6929 consensus_queue.cc:237] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:41.625432 12873 consensus_queue.cc:1035] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.627694 12822 tablet_copy_client.cc:783] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
W20240503 14:13:41.627792 6865 consensus_peers.cc:485] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9027b8bfee0443aab7420df3f7257bcb. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:41.628129 12822 tablet_copy_client.cc:647] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:41.629292 7188 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:41.630128 9529 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:41.633173 12731 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:41.634717 12822 tablet_copy_client.cc:523] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:41.638347 12822 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:41.641018 12922 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:41.646529 12411 consensus_queue.cc:1035] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:41.648922 12922 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:41.649744 7190 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:41.663290 6651 catalog_manager.cc:5484] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.676576 9529 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:41.751446 12822 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:41.752025 12822 tablet_bootstrap.cc:492] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:41.752276 12822 ts_tablet_manager.cc:1367] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.114s user 0.052s sys 0.003s
I20240503 14:13:41.754979 7190 consensus_queue.cc:237] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
W20240503 14:13:41.757577 7127 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8aaca37078ac4c4eb7f0bb6df0654860. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:41.757747 9798 raft_consensus.cc:1261] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:41.758302 9663 raft_consensus.cc:1261] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:41.758844 12411 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:41.759351 12411 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:41.759827 12822 raft_consensus.cc:348] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:41.760056 12822 raft_consensus.cc:729] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:41.760262 12822 consensus_queue.cc:260] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:41.760869 12822 ts_tablet_manager.cc:1398] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 14:13:41.773581 12877 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:41.777928 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-1d3ec4f0833d4d489f774757312efb32 received from {username='slave'} at 127.0.186.6:48891
I20240503 14:13:41.778076 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-1d3ec4f0833d4d489f774757312efb32 on tablet 1d3ec4f0833d4d489f774757312efb32 with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:41.781177 12633 raft_consensus.cc:482] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:41.781345 12633 raft_consensus.cc:504] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.782056 12633 leader_election.cc:288] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:41.783972 9663 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:41.785815 9529 raft_consensus.cc:1203] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:41.787369 6651 catalog_manager.cc:5484] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:41.789348 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b9a4e53aa24320bccf33a44ca4818b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:41.789546 7056 raft_consensus.cc:2454] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:41.790540 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b9a4e53aa24320bccf33a44ca4818b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:41.790820 9930 raft_consensus.cc:2454] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:41.793241 9733 leader_election.cc:302] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:41.796151 12633 raft_consensus.cc:2781] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:41.796310 12633 raft_consensus.cc:482] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:41.796387 12633 raft_consensus.cc:3037] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.797267 12731 raft_consensus.cc:1050] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:41.798054 12731 consensus_queue.cc:237] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:41.800303 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:41.803325 9663 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:41.804051 12922 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:41.810272 7056 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:41.811365 12922 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:41.812139 9529 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:41.813184 12728 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:41.817445 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 1708b19d3c5c4d308f4358bd42182c80 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.817947 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708b19d3c5c4d308f4358bd42182c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.818040 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 1708b19d3c5c4d308f4358bd42182c80 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.818648 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708b19d3c5c4d308f4358bd42182c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.819231 9644 tablet_service.cc:1467] Processing CreateTablet for tablet f7d921b017694f30af3e575690de44f8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.819685 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d921b017694f30af3e575690de44f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.824699 7171 tablet_service.cc:1467] Processing CreateTablet for tablet f7d921b017694f30af3e575690de44f8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.825294 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d921b017694f30af3e575690de44f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.825248 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 1708b19d3c5c4d308f4358bd42182c80 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.825809 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708b19d3c5c4d308f4358bd42182c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.828301 6911 tablet_service.cc:1467] Processing CreateTablet for tablet f7d921b017694f30af3e575690de44f8 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.828543 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 8960e39be72c4982bb80e54c47074b1b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.828946 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7d921b017694f30af3e575690de44f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.829032 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8960e39be72c4982bb80e54c47074b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.834936 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 8960e39be72c4982bb80e54c47074b1b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.835472 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8960e39be72c4982bb80e54c47074b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.844571 6909 tablet_service.cc:1467] Processing CreateTablet for tablet 8960e39be72c4982bb80e54c47074b1b (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:41.845113 6909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8960e39be72c4982bb80e54c47074b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:41.852208 7190 raft_consensus.cc:1203] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:41.853224 12914 tablet_bootstrap.cc:492] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:41.855347 12914 tablet_bootstrap.cc:654] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.859711 9663 raft_consensus.cc:1203] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 14:13:41.871529 12969 tablet_bootstrap.cc:492] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:41.873193 12915 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:41.874213 12922 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:41.875098 12915 tablet_bootstrap.cc:654] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.875433 12969 tablet_bootstrap.cc:654] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.880441 12969 tablet_bootstrap.cc:492] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:41.881325 12969 ts_tablet_manager.cc:1367] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240503 14:13:41.881008 12633 raft_consensus.cc:504] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.882377 12969 raft_consensus.cc:348] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:41.882727 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b9a4e53aa24320bccf33a44ca4818b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:41.882956 9930 raft_consensus.cc:3037] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.882781 9533 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:41.885879 9930 raft_consensus.cc:2454] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:41.886596 9733 leader_election.cc:302] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:41.886981 12853 raft_consensus.cc:2781] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:41.887140 12853 raft_consensus.cc:686] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:41.887387 12853 consensus_queue.cc:237] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:41.887382 12969 raft_consensus.cc:374] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.887732 12969 raft_consensus.cc:729] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.887951 12969 consensus_queue.cc:260] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:41.891295 6650 catalog_manager.cc:5484] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 7 to 8, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:41.898063 12914 tablet_bootstrap.cc:492] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:41.898350 12914 ts_tablet_manager.cc:1367] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.006s
I20240503 14:13:41.900151 12914 raft_consensus.cc:348] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:41.900492 12914 raft_consensus.cc:374] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.900632 12914 raft_consensus.cc:729] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.900982 12914 consensus_queue.cc:260] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:41.903306 12914 ts_tablet_manager.cc:1398] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:41.903201 7057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b9a4e53aa24320bccf33a44ca4818b" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:41.903445 7057 raft_consensus.cc:3037] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:41.905886 7057 raft_consensus.cc:2454] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:41.910697 9663 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:41.913677 12969 ts_tablet_manager.cc:1398] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20240503 14:13:41.919034 12633 leader_election.cc:288] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:41.922129 12977 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:41.923363 12970 tablet_bootstrap.cc:492] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:41.925112 12977 tablet_bootstrap.cc:654] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.926045 12970 tablet_bootstrap.cc:654] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.926256 12969 tablet_bootstrap.cc:492] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
W20240503 14:13:41.927852 6996 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 70795.5us
I20240503 14:13:41.928066 12969 tablet_bootstrap.cc:654] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.932721 12915 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:41.932901 12915 ts_tablet_manager.cc:1367] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.008s
I20240503 14:13:41.933691 12915 raft_consensus.cc:348] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:41.933917 12915 raft_consensus.cc:374] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.933975 12915 raft_consensus.cc:729] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.934186 12915 consensus_queue.cc:260] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:41.935926 12977 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:41.936105 12977 ts_tablet_manager.cc:1367] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20240503 14:13:41.937997 6929 consensus_queue.cc:237] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:41.938654 9798 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:41.939429 12915 ts_tablet_manager.cc:1398] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:41.941363 12977 raft_consensus.cc:348] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:41.941691 12977 raft_consensus.cc:374] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.941815 12977 raft_consensus.cc:729] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.941881 12969 tablet_bootstrap.cc:492] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:41.942044 12969 ts_tablet_manager.cc:1367] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20240503 14:13:41.942771 12969 raft_consensus.cc:348] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:41.942976 12969 raft_consensus.cc:374] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.943028 12969 raft_consensus.cc:729] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.943285 9663 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:41.943267 12969 consensus_queue.cc:260] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:41.943910 12977 consensus_queue.cc:260] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:41.944818 12977 ts_tablet_manager.cc:1398] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 14:13:41.944732 12728 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:41.945132 12977 tablet_bootstrap.cc:492] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:41.946862 12977 tablet_bootstrap.cc:654] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.947324 12731 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:41.948268 9663 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:41.952255 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:41.953908 9531 raft_consensus.cc:1261] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:41.954470 12970 tablet_bootstrap.cc:492] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:41.954710 12970 ts_tablet_manager.cc:1367] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.000s
I20240503 14:13:41.955902 12970 raft_consensus.cc:348] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:41.956264 12970 raft_consensus.cc:374] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.956478 6652 catalog_manager.cc:5484] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 8 to 9, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.957445 12970 raft_consensus.cc:729] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.957837 12970 consensus_queue.cc:260] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:41.958608 12970 ts_tablet_manager.cc:1398] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:41.961606 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:41.962309 12976 consensus_queue.cc:1035] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:41.963366 12969 ts_tablet_manager.cc:1398] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20240503 14:13:41.972249 12969 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:41.973994 12977 tablet_bootstrap.cc:492] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:41.974179 12977 ts_tablet_manager.cc:1367] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20240503 14:13:41.974383 9529 raft_consensus.cc:2932] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:41.975011 12977 raft_consensus.cc:348] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:41.975258 12977 raft_consensus.cc:374] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.975350 12977 raft_consensus.cc:729] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.975661 12977 consensus_queue.cc:260] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:41.977592 12977 ts_tablet_manager.cc:1398] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:41.979557 12969 tablet_bootstrap.cc:654] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.986541 6652 catalog_manager.cc:5484] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:41.988245 12982 tablet_replica.cc:330] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:41.988487 12982 raft_consensus.cc:2229] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:41.988655 12982 pending_rounds.cc:62] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: Trying to abort 1 pending ops.
I20240503 14:13:41.987816 12969 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:41.988870 12982 pending_rounds.cc:68] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7023598066882777088 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "b3e2d7664e6c45d69f4f8b20dcf0e4d0" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } } }
I20240503 14:13:41.989352 12982 raft_consensus.cc:2894] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240503 14:13:41.989473 12982 raft_consensus.cc:2866] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:41.989552 12982 raft_consensus.cc:2258] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:41.989552 12964 tablet_bootstrap.cc:492] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:41.990456 12969 ts_tablet_manager.cc:1367] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240503 14:13:41.991302 12969 raft_consensus.cc:348] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:41.991695 12969 raft_consensus.cc:374] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:41.991833 12969 raft_consensus.cc:729] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:41.992118 12969 consensus_queue.cc:260] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:41.992787 12964 tablet_bootstrap.cc:654] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:41.992769 12982 ts_tablet_manager.cc:1875] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:41.995327 12969 ts_tablet_manager.cc:1398] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:42.006747 12964 tablet_bootstrap.cc:492] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:42.007045 12964 ts_tablet_manager.cc:1367] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 14:13:42.007932 12964 raft_consensus.cc:348] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:42.008235 12964 raft_consensus.cc:374] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.008365 12964 raft_consensus.cc:729] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.008687 12964 consensus_queue.cc:260] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:42.009968 12982 ts_tablet_manager.cc:1888] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:13:42.010408 12982 log.cc:1196] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/b3e2d7664e6c45d69f4f8b20dcf0e4d0
I20240503 14:13:42.011194 12964 ts_tablet_manager.cc:1398] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:42.013258 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet b3e2d7664e6c45d69f4f8b20dcf0e4d0 (table rebalance_test_table_9 [id=a6005074a463440d9161cbe16a2ca1c9]) successfully deleted
I20240503 14:13:42.018771 12877 raft_consensus.cc:482] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.018954 12877 raft_consensus.cc:504] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:42.019912 12877 leader_election.cc:288] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:42.020419 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708b19d3c5c4d308f4358bd42182c80" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:42.020422 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708b19d3c5c4d308f4358bd42182c80" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:42.020625 6929 raft_consensus.cc:2454] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:42.020671 7056 raft_consensus.cc:2454] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:42.021063 7127 leader_election.cc:302] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:42.021453 12877 raft_consensus.cc:2781] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.021598 12877 raft_consensus.cc:482] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.021677 12877 raft_consensus.cc:3037] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.024554 12877 raft_consensus.cc:504] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:42.025283 12877 leader_election.cc:288] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:42.025748 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708b19d3c5c4d308f4358bd42182c80" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:42.025993 7056 raft_consensus.cc:3037] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.026458 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1708b19d3c5c4d308f4358bd42182c80" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:42.026644 6929 raft_consensus.cc:3037] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.029299 6929 raft_consensus.cc:2454] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:42.030905 7125 leader_election.cc:302] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:42.030972 7056 raft_consensus.cc:2454] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:42.031193 12877 raft_consensus.cc:2781] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.031289 12877 raft_consensus.cc:686] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:42.031503 12877 consensus_queue.cc:237] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:42.039610 12994 ts_tablet_manager.cc:902] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:42.043831 6652 catalog_manager.cc:5484] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.051359 12994 tablet_copy_client.cc:308] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.6:33971
I20240503 14:13:42.052235 9685 tablet_copy_service.cc:140] P f03dc37fe3234033bc665bed4f1b51f5: Received BeginTabletCopySession request for tablet 458ebc1295ee49999586eb699a29bbf8 from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:35985)
I20240503 14:13:42.052438 9685 tablet_copy_service.cc:161] P f03dc37fe3234033bc665bed4f1b51f5: Beginning new tablet copy session on tablet 458ebc1295ee49999586eb699a29bbf8 from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:35985: session id = 09723f236d8044538d920a37b5d9aa8c-458ebc1295ee49999586eb699a29bbf8
I20240503 14:13:42.055308 9685 tablet_copy_source_session.cc:205] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:42.056807 12994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 458ebc1295ee49999586eb699a29bbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.063321 12994 tablet_copy_client.cc:783] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:42.063560 12994 tablet_copy_client.cc:647] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:42.065307 12994 tablet_copy_client.cc:523] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:42.070979 12994 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:42.073362 12961 raft_consensus.cc:482] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.073527 12961 raft_consensus.cc:504] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:42.074196 12961 leader_election.cc:288] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:42.075018 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d921b017694f30af3e575690de44f8" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:42.075035 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d921b017694f30af3e575690de44f8" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:42.075229 7190 raft_consensus.cc:2454] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:42.075271 9663 raft_consensus.cc:2454] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:42.075881 6866 leader_election.cc:302] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:42.076817 12961 raft_consensus.cc:2781] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.076905 12961 raft_consensus.cc:482] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.076979 12961 raft_consensus.cc:3037] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.079610 12961 raft_consensus.cc:504] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:42.081121 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d921b017694f30af3e575690de44f8" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:42.081342 9663 raft_consensus.cc:3037] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.082938 12994 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:42.083448 12994 tablet_bootstrap.cc:492] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:42.083438 9663 raft_consensus.cc:2454] T f7d921b017694f30af3e575690de44f8 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:42.083807 12994 ts_tablet_manager.cc:1367] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240503 14:13:42.083950 6865 leader_election.cc:302] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:42.084014 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7d921b017694f30af3e575690de44f8" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:42.084204 7190 raft_consensus.cc:3037] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.084743 12994 raft_consensus.cc:348] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:42.085028 12994 raft_consensus.cc:729] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:42.085299 12994 consensus_queue.cc:260] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:42.086040 12994 ts_tablet_manager.cc:1398] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:42.086488 7190 raft_consensus.cc:2454] T f7d921b017694f30af3e575690de44f8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:42.086946 12976 raft_consensus.cc:2781] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.087074 12976 raft_consensus.cc:686] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:42.087301 9685 tablet_copy_service.cc:342] P f03dc37fe3234033bc665bed4f1b51f5: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-458ebc1295ee49999586eb699a29bbf8 received from {username='slave'} at 127.0.186.3:35985
I20240503 14:13:42.087428 9685 tablet_copy_service.cc:433] P f03dc37fe3234033bc665bed4f1b51f5: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-458ebc1295ee49999586eb699a29bbf8 on tablet 458ebc1295ee49999586eb699a29bbf8 with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:42.087320 12976 consensus_queue.cc:237] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:42.089190 12961 leader_election.cc:288] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:42.092492 6652 catalog_manager.cc:5484] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.148339 6652 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:42.155965 7056 raft_consensus.cc:1261] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:42.157935 12633 consensus_queue.cc:1035] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.159513 9929 raft_consensus.cc:1261] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:42.161125 12853 consensus_queue.cc:1035] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.179279 11977 raft_consensus.cc:1050] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:42.180321 11977 consensus_queue.cc:237] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:42.193236 12961 raft_consensus.cc:482] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:42.193390 12961 raft_consensus.cc:504] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:42.194046 12961 leader_election.cc:288] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:42.194685 9529 raft_consensus.cc:1261] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:42.194715 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960e39be72c4982bb80e54c47074b1b" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:42.194892 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960e39be72c4982bb80e54c47074b1b" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:42.194948 9928 raft_consensus.cc:2454] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:42.195115 9533 raft_consensus.cc:2454] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:42.195343 6865 leader_election.cc:302] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:42.195504 12313 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:42.195721 12961 raft_consensus.cc:2781] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:42.195802 12961 raft_consensus.cc:482] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:42.195868 12961 raft_consensus.cc:3037] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.198534 12961 raft_consensus.cc:504] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:42.200105 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960e39be72c4982bb80e54c47074b1b" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:42.200366 9928 raft_consensus.cc:3037] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.200456 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8960e39be72c4982bb80e54c47074b1b" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:42.200717 9533 raft_consensus.cc:3037] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:42.203845 9533 raft_consensus.cc:2454] T 8960e39be72c4982bb80e54c47074b1b P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:42.204233 6867 leader_election.cc:302] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12; no voters:
I20240503 14:13:42.204520 12728 raft_consensus.cc:2781] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:42.204624 12728 raft_consensus.cc:686] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:42.204885 12728 consensus_queue.cc:237] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:42.205060 9928 raft_consensus.cc:2454] T 8960e39be72c4982bb80e54c47074b1b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:42.207566 12961 leader_election.cc:288] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:42.218894 6652 catalog_manager.cc:5484] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.226711 13034 ts_tablet_manager.cc:902] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:42.229310 13034 tablet_copy_client.cc:308] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:42.230810 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 8aaca37078ac4c4eb7f0bb6df0654860 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:48297)
I20240503 14:13:42.231029 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 8aaca37078ac4c4eb7f0bb6df0654860 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:48297: session id = 5b7e36bf354f480c90859cb8867b59cf-8aaca37078ac4c4eb7f0bb6df0654860
I20240503 14:13:42.234447 7211 tablet_copy_source_session.cc:205] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:42.237170 9798 raft_consensus.cc:1261] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:42.237846 13040 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:42.240994 6799 raft_consensus.cc:1261] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:42.243484 12820 raft_consensus.cc:1050] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
I20240503 14:13:42.244696 12820 consensus_queue.cc:237] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:42.248206 13053 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:42.251397 6652 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 76b9a4e53aa24320bccf33a44ca4818b (table rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12]). Expected version 1 got 0
I20240503 14:13:42.251721 13053 tablet.cc:1620] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:42.260625 12959 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:42.272450 13035 ts_tablet_manager.cc:902] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:42.281622 6651 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 76b9a4e53aa24320bccf33a44ca4818b (table rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12]). Expected version 1 got 0
I20240503 14:13:42.281848 7190 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:42.283402 9533 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:42.283478 9930 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:42.284297 12924 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:42.284941 12924 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:42.291220 12924 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:42.294485 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12] alter complete (version 1)
I20240503 14:13:42.300427 12313 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:42.304200 9531 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:42.311560 6799 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:42.313256 6651 catalog_manager.cc:5484] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:42.330942 9929 consensus_queue.cc:237] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:42.331952 9798 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:42.339152 6799 raft_consensus.cc:1261] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:42.340462 12313 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:42.336158 13034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aaca37078ac4c4eb7f0bb6df0654860. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.337407 9529 raft_consensus.cc:1261] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:42.343284 12313 consensus_queue.cc:1035] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:42.345528 12959 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:42.346966 12820 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:42.349967 13062 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:42.350078 6798 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:42.350584 13062 tablet.cc:1620] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:42.353466 9529 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:42.354154 6650 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 76b9a4e53aa24320bccf33a44ca4818b (table rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12]). Expected version 1 got 0
I20240503 14:13:42.356354 13063 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:42.355391 9929 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:42.356953 13063 tablet.cc:1620] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:42.356788 7190 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:42.359294 6650 catalog_manager.cc:5484] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:42.366415 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8811b896c0504775b03baf4925026adf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:42.367030 6652 catalog_manager.cc:5484] T 8811b896c0504775b03baf4925026adf P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:42.377287 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 8811b896c0504775b03baf4925026adf with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:42.381310 9663 consensus_queue.cc:237] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:42.383468 9531 raft_consensus.cc:2932] T 8811b896c0504775b03baf4925026adf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:42.384436 9530 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:42.385504 12924 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:42.385848 12813 raft_consensus.cc:1050] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:42.386920 12813 consensus_queue.cc:237] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:42.387666 13068 tablet_replica.cc:330] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:42.387897 13068 raft_consensus.cc:2229] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:42.388160 13068 raft_consensus.cc:2258] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.392110 6651 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 76b9a4e53aa24320bccf33a44ca4818b (table rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12]). Expected version 1 got 0
I20240503 14:13:42.395357 13068 ts_tablet_manager.cc:1875] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:42.404356 7190 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:42.407012 9663 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:42.407580 12805 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:42.409361 12820 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:42.411202 9533 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:42.411396 6929 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:42.411281 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:42.411998 12966 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:42.412631 12966 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:42.421517 12966 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:42.426411 13053 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:42.426954 13053 tablet.cc:1620] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: MemRowSet was empty: no flush needed.
I20240503 14:13:42.427516 13063 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:42.428225 13063 tablet.cc:1620] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:42.432585 9533 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:42.436839 6650 catalog_manager.cc:5484] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 8 to 9, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:42.444885 6931 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:42.448555 7056 consensus_queue.cc:237] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:42.452073 9533 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:42.452580 12805 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:42.462772 6652 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12] alter complete (version 2)
I20240503 14:13:42.466192 12966 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:42.471439 13035 tablet_copy_client.cc:308] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:42.472200 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 9027b8bfee0443aab7420df3f7257bcb from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:39445)
I20240503 14:13:42.472366 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 9027b8bfee0443aab7420df3f7257bcb from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:39445: session id = ea801601713a4d55aee0de9e3473e0bc-9027b8bfee0443aab7420df3f7257bcb
I20240503 14:13:42.472631 9665 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:42.475320 6951 tablet_copy_source_session.cc:205] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:42.476835 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d3ec4f0833d4d489f774757312efb32 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:42.478331 9531 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:42.479476 6650 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 76b9a4e53aa24320bccf33a44ca4818b (table rebalancer_dynamic_table [id=5ca8fb413554489c964f8bf478163f12]). Expected version 2 got 1
I20240503 14:13:42.481226 13062 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:42.481885 13062 tablet.cc:1620] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:42.485852 9662 raft_consensus.cc:1261] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:42.486160 13035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9027b8bfee0443aab7420df3f7257bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.488516 6650 catalog_manager.cc:5484] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 9 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:42.489326 12966 consensus_queue.cc:1035] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:42.493986 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d3ec4f0833d4d489f774757312efb32 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:42.495292 13068 ts_tablet_manager.cc:1888] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:42.495445 13068 log.cc:1196] T 8811b896c0504775b03baf4925026adf P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/8811b896c0504775b03baf4925026adf
I20240503 14:13:42.496168 12961 consensus_queue.cc:1035] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.496296 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 8811b896c0504775b03baf4925026adf (table rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]) successfully deleted
I20240503 14:13:42.503018 13035 tablet_copy_client.cc:783] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:42.503376 13035 tablet_copy_client.cc:647] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:42.509088 13035 tablet_copy_client.cc:523] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:42.510095 12924 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:42.513682 13035 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:42.518098 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 603002fdf6b84d3ea63c65636eb42cb1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:42.518095 7191 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:42.518571 13078 tablet_replica.cc:330] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:42.518819 13078 raft_consensus.cc:2229] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:42.519279 13078 raft_consensus.cc:2258] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.519434 9531 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:42.494239 9663 raft_consensus.cc:2932] T 1d3ec4f0833d4d489f774757312efb32 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:42.525976 6651 catalog_manager.cc:5484] T 603002fdf6b84d3ea63c65636eb42cb1 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:42.430691 13034 tablet_copy_client.cc:783] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:42.528764 13034 tablet_copy_client.cc:647] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:42.531538 13078 ts_tablet_manager.cc:1875] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:42.532972 13034 tablet_copy_client.cc:523] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:42.538434 13078 ts_tablet_manager.cc:1888] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:13:42.539219 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 603002fdf6b84d3ea63c65636eb42cb1 with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:42.547701 13034 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:42.548074 13078 log.cc:1196] T 1d3ec4f0833d4d489f774757312efb32 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/1d3ec4f0833d4d489f774757312efb32
I20240503 14:13:42.549038 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 1d3ec4f0833d4d489f774757312efb32 (table rebalance_test_table_7 [id=66a7e23023c0463a8677a94e3558b695]) successfully deleted
I20240503 14:13:42.554117 12976 consensus_queue.cc:1035] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.554953 13081 tablet_replica.cc:330] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:42.555943 13081 raft_consensus.cc:2229] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:42.557690 13081 raft_consensus.cc:2258] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.559103 13081 ts_tablet_manager.cc:1875] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:42.574519 6652 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:42.576117 12873 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.579581 13034 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:42.580150 13034 tablet_bootstrap.cc:492] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:42.580449 13034 ts_tablet_manager.cc:1367] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.007s
I20240503 14:13:42.581811 13034 raft_consensus.cc:348] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:42.582187 13034 raft_consensus.cc:729] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:42.582479 13034 consensus_queue.cc:260] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:42.587246 13034 ts_tablet_manager.cc:1398] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:13:42.588074 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-8aaca37078ac4c4eb7f0bb6df0654860 received from {username='slave'} at 127.0.186.5:48297
I20240503 14:13:42.588207 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-8aaca37078ac4c4eb7f0bb6df0654860 on tablet 8aaca37078ac4c4eb7f0bb6df0654860 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:42.593714 12961 consensus_queue.cc:1035] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.595755 13053 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:42.600292 13063 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:42.608704 13062 tablet.cc:1722] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:42.612813 7057 raft_consensus.cc:1203] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:42.628342 12411 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.633538 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=5ca8fb413554489c964f8bf478163f12] alter complete (version 3)
I20240503 14:13:42.668591 12728 consensus_queue.cc:1035] T 8960e39be72c4982bb80e54c47074b1b P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:42.679847 13081 ts_tablet_manager.cc:1888] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:42.680104 13081 log.cc:1196] T 603002fdf6b84d3ea63c65636eb42cb1 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/603002fdf6b84d3ea63c65636eb42cb1
I20240503 14:13:42.682209 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 603002fdf6b84d3ea63c65636eb42cb1 (table rebalancer_extra_table_24 [id=5da909ed6fb54945bb23595d345d9032]) successfully deleted
I20240503 14:13:42.699065 9533 raft_consensus.cc:1203] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:42.736904 6658 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:42.737201 6658 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:42.740998 6658 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 76b9a4e53aa24320bccf33a44ca4818b
I20240503 14:13:42.742236 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b9a4e53aa24320bccf33a44ca4818b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:42 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:42.742306 9910 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b9a4e53aa24320bccf33a44ca4818b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:42 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:42.742666 13081 tablet_replica.cc:330] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:42.742795 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b9a4e53aa24320bccf33a44ca4818b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:42 UTC) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:42.742990 13081 raft_consensus.cc:2229] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:42.743108 13068 tablet_replica.cc:330] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:42.743229 13081 raft_consensus.cc:2258] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.743342 13068 raft_consensus.cc:2229] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:42.743826 13068 raft_consensus.cc:2258] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.744666 13081 ts_tablet_manager.cc:1875] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:42.745153 13068 ts_tablet_manager.cc:1875] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:42.756047 13068 ts_tablet_manager.cc:1888] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:42.756657 13081 ts_tablet_manager.cc:1888] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:42.756857 13081 log.cc:1196] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/76b9a4e53aa24320bccf33a44ca4818b
I20240503 14:13:42.758373 13068 log.cc:1196] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/76b9a4e53aa24320bccf33a44ca4818b
I20240503 14:13:42.759159 13068 ts_tablet_manager.cc:1909] T 76b9a4e53aa24320bccf33a44ca4818b P 3efdbd3506274dba846ae40404f0572c: Deleting consensus metadata
I20240503 14:13:42.770181 13081 ts_tablet_manager.cc:1909] T 76b9a4e53aa24320bccf33a44ca4818b P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:42.771986 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 76b9a4e53aa24320bccf33a44ca4818b (table rebalancer_dynamic_table_new_name [id=5ca8fb413554489c964f8bf478163f12]) successfully deleted
I20240503 14:13:42.774847 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 76b9a4e53aa24320bccf33a44ca4818b (table rebalancer_dynamic_table_new_name [id=5ca8fb413554489c964f8bf478163f12]) successfully deleted
I20240503 14:13:42.776348 13098 tablet_replica.cc:330] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:42.782913 13098 raft_consensus.cc:2229] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:42.783241 13098 raft_consensus.cc:2258] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:42.789602 13035 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:42.790541 13098 ts_tablet_manager.cc:1875] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:42.792431 13035 tablet_bootstrap.cc:492] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:42.792853 13035 ts_tablet_manager.cc:1367] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.279s user 0.056s sys 0.005s
I20240503 14:13:42.793900 13035 raft_consensus.cc:348] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:42.799364 13035 raft_consensus.cc:729] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:42.799865 13035 consensus_queue.cc:260] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:42.800688 13098 ts_tablet_manager.cc:1888] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:42.805716 13035 ts_tablet_manager.cc:1398] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 14:13:42.810623 6651 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:42.811256 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-9027b8bfee0443aab7420df3f7257bcb received from {username='slave'} at 127.0.186.8:39445
I20240503 14:13:42.811403 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-9027b8bfee0443aab7420df3f7257bcb on tablet 9027b8bfee0443aab7420df3f7257bcb with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:42.816663 13098 log.cc:1196] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/76b9a4e53aa24320bccf33a44ca4818b
I20240503 14:13:42.818696 13098 ts_tablet_manager.cc:1909] T 76b9a4e53aa24320bccf33a44ca4818b P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:42.826928 9777 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc7cefd974149148dcf856a50ea168c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.826894 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc7cefd974149148dcf856a50ea168c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.827427 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc7cefd974149148dcf856a50ea168c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.827648 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc7cefd974149148dcf856a50ea168c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.826928 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 964b1598fb17446dacfe2b5931df5d41 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.828291 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964b1598fb17446dacfe2b5931df5d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.832130 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 964b1598fb17446dacfe2b5931df5d41 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.832167 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 964b1598fb17446dacfe2b5931df5d41 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.832566 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964b1598fb17446dacfe2b5931df5d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.832746 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 964b1598fb17446dacfe2b5931df5d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.834995 9910 tablet_service.cc:1467] Processing CreateTablet for tablet 5dc7cefd974149148dcf856a50ea168c (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.838707 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 27b495c1b7df4f4391820da60484f881 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.839188 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b495c1b7df4f4391820da60484f881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.839838 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 27b495c1b7df4f4391820da60484f881 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.840281 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b495c1b7df4f4391820da60484f881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.839756 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 27b495c1b7df4f4391820da60484f881 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:42.847352 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:42.848800 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc7cefd974149148dcf856a50ea168c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.849766 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27b495c1b7df4f4391820da60484f881. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.863580 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 76b9a4e53aa24320bccf33a44ca4818b (table rebalancer_dynamic_table_new_name [id=5ca8fb413554489c964f8bf478163f12]) successfully deleted
I20240503 14:13:42.878769 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 325917993a6c4453a45f733957a683a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82]), partition=
I20240503 14:13:42.879201 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325917993a6c4453a45f733957a683a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.879951 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 325917993a6c4453a45f733957a683a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82]), partition=
I20240503 14:13:42.880326 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325917993a6c4453a45f733957a683a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.883154 13106 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:42.888206 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 325917993a6c4453a45f733957a683a8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82]), partition=
I20240503 14:13:42.888715 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 325917993a6c4453a45f733957a683a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:42.895495 13106 tablet_bootstrap.cc:654] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.901685 13105 tablet_bootstrap.cc:492] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:42.903617 13105 tablet_bootstrap.cc:654] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.907269 13107 tablet_bootstrap.cc:492] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:42.908829 13108 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:42.909624 13109 tablet_bootstrap.cc:492] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:42.912014 13107 tablet_bootstrap.cc:654] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.917559 13106 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:42.917904 13109 tablet_bootstrap.cc:654] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.918438 13106 ts_tablet_manager.cc:1367] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.000s
I20240503 14:13:42.920312 13106 raft_consensus.cc:348] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:42.920696 13106 raft_consensus.cc:374] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.922101 13103 tablet_bootstrap.cc:492] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:42.922531 13106 raft_consensus.cc:729] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.922813 13106 consensus_queue.cc:260] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:42.925652 13104 tablet_bootstrap.cc:492] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:42.929512 13108 tablet_bootstrap.cc:654] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.932829 13109 tablet_bootstrap.cc:492] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:42.933108 13109 ts_tablet_manager.cc:1367] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240503 14:13:42.933450 13104 tablet_bootstrap.cc:654] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.934001 13109 raft_consensus.cc:348] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:42.934373 13109 raft_consensus.cc:374] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.934466 13109 raft_consensus.cc:729] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.934731 13109 consensus_queue.cc:260] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:42.935909 13106 ts_tablet_manager.cc:1398] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20240503 14:13:42.940079 13105 tablet_bootstrap.cc:492] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:42.940320 13105 ts_tablet_manager.cc:1367] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240503 14:13:42.940645 13109 ts_tablet_manager.cc:1398] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:13:42.941095 13106 tablet_bootstrap.cc:492] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:42.944054 13105 raft_consensus.cc:348] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:42.944474 13105 raft_consensus.cc:374] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.944623 13105 raft_consensus.cc:729] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.944972 13105 consensus_queue.cc:260] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:42.948729 13106 tablet_bootstrap.cc:654] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.948755 13107 tablet_bootstrap.cc:492] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:42.948926 13107 ts_tablet_manager.cc:1367] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.007s
I20240503 14:13:42.951304 13105 ts_tablet_manager.cc:1398] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:42.947373 13103 tablet_bootstrap.cc:654] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.954097 13104 tablet_bootstrap.cc:492] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:42.954815 13104 ts_tablet_manager.cc:1367] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240503 14:13:42.955766 13104 raft_consensus.cc:348] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:42.956126 13104 raft_consensus.cc:374] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.956235 13104 raft_consensus.cc:729] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.956511 13104 consensus_queue.cc:260] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:42.958005 13107 raft_consensus.cc:348] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:42.958312 13107 raft_consensus.cc:374] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.958658 13107 raft_consensus.cc:729] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.960106 13104 ts_tablet_manager.cc:1398] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 14:13:42.960539 13104 tablet_bootstrap.cc:492] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:42.961244 13107 consensus_queue.cc:260] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:42.962056 13107 ts_tablet_manager.cc:1398] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 14:13:42.962404 13107 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:42.962411 13104 tablet_bootstrap.cc:654] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.966102 13106 tablet_bootstrap.cc:492] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:42.966341 13106 ts_tablet_manager.cc:1367] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.010s
I20240503 14:13:42.967468 13108 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:42.967362 13106 raft_consensus.cc:348] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:42.967674 13106 raft_consensus.cc:374] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.967794 13108 ts_tablet_manager.cc:1367] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.006s
I20240503 14:13:42.967839 13106 raft_consensus.cc:729] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.968314 13106 consensus_queue.cc:260] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:42.968864 13108 raft_consensus.cc:348] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:42.969107 13108 raft_consensus.cc:374] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.969233 13108 raft_consensus.cc:729] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.969578 13108 consensus_queue.cc:260] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:42.973528 13108 ts_tablet_manager.cc:1398] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 14:13:42.973950 13108 tablet_bootstrap.cc:492] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:42.976614 13108 tablet_bootstrap.cc:654] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.979997 13106 ts_tablet_manager.cc:1398] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240503 14:13:42.986078 13107 tablet_bootstrap.cc:654] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:42.987550 13103 tablet_bootstrap.cc:492] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:42.988258 13103 ts_tablet_manager.cc:1367] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.000s
I20240503 14:13:42.988844 13108 tablet_bootstrap.cc:492] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:42.989069 13108 ts_tablet_manager.cc:1367] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20240503 14:13:42.989511 13103 raft_consensus.cc:348] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:42.990164 13103 raft_consensus.cc:374] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.990329 13103 raft_consensus.cc:729] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.990797 13103 consensus_queue.cc:260] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:42.991395 13104 tablet_bootstrap.cc:492] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:42.991644 13104 ts_tablet_manager.cc:1367] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.005s
I20240503 14:13:42.990072 13108 raft_consensus.cc:348] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:42.991978 13108 raft_consensus.cc:374] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.992125 13108 raft_consensus.cc:729] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.992463 13108 consensus_queue.cc:260] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:42.992866 13104 raft_consensus.cc:348] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:42.993204 13104 raft_consensus.cc:374] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:42.993292 13108 ts_tablet_manager.cc:1398] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:42.993330 13104 raft_consensus.cc:729] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:42.993676 13104 consensus_queue.cc:260] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:42.994421 13104 ts_tablet_manager.cc:1398] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 14:13:42.994808 13104 tablet_bootstrap.cc:492] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:42.996752 13104 tablet_bootstrap.cc:654] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.005209 13104 tablet_bootstrap.cc:492] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:43.005445 13104 ts_tablet_manager.cc:1367] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 14:13:43.013669 13103 ts_tablet_manager.cc:1398] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 14:13:43.017926 13104 raft_consensus.cc:348] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.018364 13104 raft_consensus.cc:374] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.018497 13104 raft_consensus.cc:729] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.018841 13104 consensus_queue.cc:260] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.019568 13104 ts_tablet_manager.cc:1398] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240503 14:13:43.028841 13107 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:43.029444 13107 ts_tablet_manager.cc:1367] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.004s
I20240503 14:13:43.030238 13107 raft_consensus.cc:348] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.030453 13107 raft_consensus.cc:374] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:43.030552 13107 raft_consensus.cc:729] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:43.030815 13107 consensus_queue.cc:260] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.032013 13107 ts_tablet_manager.cc:1398] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:43.039124 13127 raft_consensus.cc:482] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.039324 13127 raft_consensus.cc:504] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.040123 13127 leader_election.cc:288] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:43.040980 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc7cefd974149148dcf856a50ea168c" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:43.041266 9533 raft_consensus.cc:2454] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:43.042003 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc7cefd974149148dcf856a50ea168c" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:43.042237 9929 raft_consensus.cc:2454] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:43.042843 9733 leader_election.cc:302] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:43.043293 13127 raft_consensus.cc:2781] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.043481 13127 raft_consensus.cc:482] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.043653 13127 raft_consensus.cc:3037] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.046103 12877 raft_consensus.cc:482] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.046240 12877 raft_consensus.cc:504] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:43.046969 12877 leader_election.cc:288] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:43.047204 12858 raft_consensus.cc:482] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.047356 12858 raft_consensus.cc:504] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.048039 12858 leader_election.cc:288] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:43.048444 13127 raft_consensus.cc:504] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.048748 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964b1598fb17446dacfe2b5931df5d41" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:43.048923 6798 raft_consensus.cc:2454] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:43.049973 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc7cefd974149148dcf856a50ea168c" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:43.050300 9929 raft_consensus.cc:3037] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.051023 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964b1598fb17446dacfe2b5931df5d41" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:43.051273 9798 raft_consensus.cc:2454] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:43.051800 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325917993a6c4453a45f733957a683a8" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:43.051843 7125 leader_election.cc:302] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:43.051987 7056 raft_consensus.cc:2454] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:43.052397 6736 leader_election.cc:302] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:43.052577 12411 raft_consensus.cc:2781] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.052656 12858 raft_consensus.cc:2781] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.052728 12858 raft_consensus.cc:482] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.052702 12411 raft_consensus.cc:482] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.052798 12858 raft_consensus.cc:3037] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.052803 12411 raft_consensus.cc:3037] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.054054 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325917993a6c4453a45f733957a683a8" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:43.054339 9533 raft_consensus.cc:2454] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:43.055814 12858 raft_consensus.cc:504] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.056434 12858 leader_election.cc:288] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:43.057086 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325917993a6c4453a45f733957a683a8" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:43.057237 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "325917993a6c4453a45f733957a683a8" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:43.057317 7056 raft_consensus.cc:3037] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.057478 9533 raft_consensus.cc:3037] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.058064 12411 raft_consensus.cc:504] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:43.059518 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964b1598fb17446dacfe2b5931df5d41" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:43.059864 9798 raft_consensus.cc:3037] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.060546 9533 raft_consensus.cc:2454] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:43.060775 7056 raft_consensus.cc:2454] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:43.061020 6736 leader_election.cc:302] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:43.061303 12858 raft_consensus.cc:2781] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.064630 6799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "964b1598fb17446dacfe2b5931df5d41" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:43.064909 6799 raft_consensus.cc:3037] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.073308 9798 raft_consensus.cc:2454] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:43.074025 7126 leader_election.cc:302] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:43.074470 12877 raft_consensus.cc:2781] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.074626 12877 raft_consensus.cc:686] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:43.074949 12877 consensus_queue.cc:237] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:43.075313 6799 raft_consensus.cc:2454] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:43.080125 12411 leader_election.cc:288] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:43.081033 9929 raft_consensus.cc:2454] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:43.082398 9733 leader_election.cc:302] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:43.082859 12853 raft_consensus.cc:2781] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.083083 12853 raft_consensus.cc:686] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:43.083484 12853 consensus_queue.cc:237] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.086969 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dc7cefd974149148dcf856a50ea168c" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:43.087280 9533 raft_consensus.cc:3037] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.084638 6658 catalog_manager.cc:5484] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:43.090900 9533 raft_consensus.cc:2454] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:43.092464 13127 leader_election.cc:288] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:43.103806 12858 raft_consensus.cc:686] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:43.104249 12858 consensus_queue.cc:237] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:43.104521 6658 catalog_manager.cc:5484] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:43.111210 6658 catalog_manager.cc:5484] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:43.114240 9929 raft_consensus.cc:1203] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:43.131019 12877 raft_consensus.cc:1050] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:43.131844 12877 consensus_queue.cc:237] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:43.135715 9533 raft_consensus.cc:1261] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:43.135831 9665 raft_consensus.cc:1261] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:43.136281 9798 raft_consensus.cc:1261] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:43.136462 12873 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:43.137951 12411 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:43.141072 12873 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:43.166050 13146 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:43.167090 9531 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:43.170746 9665 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:43.174669 6658 catalog_manager.cc:5484] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:43.179805 9797 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:43.182395 13074 raft_consensus.cc:1050] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:43.183439 13074 consensus_queue.cc:237] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:43.186513 7187 consensus_queue.cc:237] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:43.189035 9665 raft_consensus.cc:1261] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:43.189656 12411 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:43.191656 9533 raft_consensus.cc:1261] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:43.192880 12877 consensus_queue.cc:1035] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:43.199031 12411 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:43.199704 9665 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:43.202818 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8aaca37078ac4c4eb7f0bb6df0654860 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:43.205423 12858 raft_consensus.cc:482] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:43.205590 12858 raft_consensus.cc:504] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:43.206248 12858 leader_election.cc:288] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:43.207031 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b495c1b7df4f4391820da60484f881" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:43.207304 6931 raft_consensus.cc:2454] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:43.207787 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b495c1b7df4f4391820da60484f881" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:43.208024 9929 raft_consensus.cc:2454] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 0.
I20240503 14:13:43.208477 6734 leader_election.cc:302] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:43.208732 6651 catalog_manager.cc:5484] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:43.209342 12858 raft_consensus.cc:2781] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:43.209424 12858 raft_consensus.cc:482] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:43.209482 12858 raft_consensus.cc:3037] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.209087 9531 raft_consensus.cc:2932] T 8aaca37078ac4c4eb7f0bb6df0654860 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:43.212035 12858 raft_consensus.cc:504] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:43.213383 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b495c1b7df4f4391820da60484f881" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:43.213666 9929 raft_consensus.cc:3037] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.216629 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 8aaca37078ac4c4eb7f0bb6df0654860 with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:43.217077 13068 tablet_replica.cc:330] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:43.217375 13068 raft_consensus.cc:2229] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:43.217691 13068 raft_consensus.cc:2258] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.219306 9929 raft_consensus.cc:2454] T 27b495c1b7df4f4391820da60484f881 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:43.221005 13068 ts_tablet_manager.cc:1875] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:43.221558 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27b495c1b7df4f4391820da60484f881" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:43.221804 6931 raft_consensus.cc:3037] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:43.224982 6931 raft_consensus.cc:2454] T 27b495c1b7df4f4391820da60484f881 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:43.225577 6734 leader_election.cc:302] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:43.225733 12858 leader_election.cc:288] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:43.225881 13135 raft_consensus.cc:2781] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:43.226037 13135 raft_consensus.cc:686] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:43.226279 13135 consensus_queue.cc:237] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:43.229137 7056 raft_consensus.cc:1261] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:43.231077 13068 ts_tablet_manager.cc:1888] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:43.231549 13068 log.cc:1196] T 8aaca37078ac4c4eb7f0bb6df0654860 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/8aaca37078ac4c4eb7f0bb6df0654860
I20240503 14:13:43.232771 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 8aaca37078ac4c4eb7f0bb6df0654860 (table rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]) successfully deleted
I20240503 14:13:43.239876 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:43.257212 9929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35998
I20240503 14:13:43.257567 9929 raft_consensus.cc:595] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:43.267813 12313 raft_consensus.cc:979] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: : Instructing follower e1bf5fb7402c46829974e2f74cd88708 to start an election
I20240503 14:13:43.268010 12313 raft_consensus.cc:1067] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Signalling peer e1bf5fb7402c46829974e2f74cd88708 to start an election
I20240503 14:13:43.270344 7187 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196"
dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
from {username='slave'} at 127.0.186.8:41897
I20240503 14:13:43.270741 7187 raft_consensus.cc:482] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:43.270943 7187 raft_consensus.cc:3037] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.274497 7187 raft_consensus.cc:504] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:43.276502 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:43.276779 9929 raft_consensus.cc:3032] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:43.276893 9929 raft_consensus.cc:729] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:43.277289 9929 consensus_queue.cc:260] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:43.277798 9929 raft_consensus.cc:3037] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.282101 9929 raft_consensus.cc:2454] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 2.
I20240503 14:13:43.284061 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:43.284273 13158 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:43.284847 6931 raft_consensus.cc:3037] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.289811 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b97ddea52b624c668cb0b28ab0f38196" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:43.288955 7187 leader_election.cc:288] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:43.290087 9663 raft_consensus.cc:3037] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.292071 6931 raft_consensus.cc:2454] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 2.
I20240503 14:13:43.292623 7125 leader_election.cc:302] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, e1bf5fb7402c46829974e2f74cd88708, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:43.293268 9663 raft_consensus.cc:2454] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 2.
I20240503 14:13:43.293429 9533 raft_consensus.cc:1261] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.294006 12411 raft_consensus.cc:2781] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:43.294150 12411 raft_consensus.cc:686] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 2 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:43.294416 13135 consensus_queue.cc:1035] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.294418 12411 consensus_queue.cc:237] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:43.301086 6658 catalog_manager.cc:5484] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:43.321074 6650 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 325917993a6c4453a45f733957a683a8 (table rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82]). Expected version 1 got 0
I20240503 14:13:43.317036 6658 catalog_manager.cc:5484] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 1 to 2, leader changed from ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 2 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:13:43.325680 9600 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.4:44337, user_credentials={real_user=slave}} blocked reactor thread for 97651.4us
I20240503 14:13:43.326478 7187 raft_consensus.cc:1261] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:43.333071 12820 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:43.338846 13074 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:43.343577 7187 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
W20240503 14:13:43.352027 6734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 98931.3us
I20240503 14:13:43.352118 13178 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.351944 6658 catalog_manager.cc:5484] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:43.352846 13178 tablet.cc:1620] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:43.360731 6799 raft_consensus.cc:1261] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:43.364089 13179 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.367925 6658 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet 325917993a6c4453a45f733957a683a8 (table rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82]). Expected version 1 got 0
I20240503 14:13:43.368558 13179 tablet.cc:1620] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:43.371615 7057 raft_consensus.cc:1261] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.375033 12858 consensus_queue.cc:1035] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.377597 13158 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:43.378829 7056 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:43.403208 9665 consensus_queue.cc:237] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:43.406545 7056 raft_consensus.cc:1261] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:43.408471 13157 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:43.418129 7059 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:43.425976 7187 raft_consensus.cc:1261] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:43.430756 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82] alter complete (version 1)
I20240503 14:13:43.433912 13158 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:43.435806 7187 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:43.444311 6658 catalog_manager.cc:5484] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 3 to 4, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:43.450114 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 458ebc1295ee49999586eb699a29bbf8 with delete_type TABLET_DATA_TOMBSTONED (TS f1a669e28f014fe287932acf56bebc13 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:43.450776 13074 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:43.454972 6799 raft_consensus.cc:2932] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:43.465451 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 458ebc1295ee49999586eb699a29bbf8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:43.473093 13185 tablet_replica.cc:330] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:43.477061 13185 raft_consensus.cc:2229] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:43.478545 13185 raft_consensus.cc:2258] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.480022 13185 ts_tablet_manager.cc:1875] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:43.486269 13189 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:43.486897 13189 tablet.cc:1620] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:43.490967 6651 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 325917993a6c4453a45f733957a683a8 (table rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82]). Expected version 1 got 0
I20240503 14:13:43.501858 13185 ts_tablet_manager.cc:1888] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:43.502152 13185 log.cc:1196] T 458ebc1295ee49999586eb699a29bbf8 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/458ebc1295ee49999586eb699a29bbf8
I20240503 14:13:43.503190 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 458ebc1295ee49999586eb699a29bbf8 (table rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]) successfully deleted
I20240503 14:13:43.505064 12853 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.507154 6931 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95"
dest_uuid: "7186ed8510424fd597173eaf10c57b12"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41004
I20240503 14:13:43.507431 6931 raft_consensus.cc:595] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:43.524479 12877 consensus_queue.cc:1035] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.530233 12878 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.558651 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:43.559248 12873 consensus_queue.cc:1035] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.574270 13178 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.574769 13178 tablet.cc:1620] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:43.574903 6658 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet 325917993a6c4453a45f733957a683a8 (table rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82]). Expected version 2 got 1
I20240503 14:13:43.576543 13179 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.577116 13179 tablet.cc:1620] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:43.578891 13189 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:43.579387 13189 tablet.cc:1620] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:43.591202 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=bfbd326ff9fe4a4f90e5601716217d82] alter complete (version 2)
I20240503 14:13:43.622669 9797 consensus_queue.cc:237] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
W20240503 14:13:43.625355 9733 consensus_peers.cc:485] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0cb6a131c94047649646ad83a0074133. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:43.625411 9533 raft_consensus.cc:1261] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.626034 9665 raft_consensus.cc:1261] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:43.628266 12878 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:43.630153 12878 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:43.631574 12853 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:43.635488 9533 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:43.640929 9665 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:43.645005 6651 catalog_manager.cc:5484] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:43.654292 12961 raft_consensus.cc:979] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: : Instructing follower f03dc37fe3234033bc665bed4f1b51f5 to start an election
I20240503 14:13:43.654446 12961 raft_consensus.cc:1067] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Signalling peer f03dc37fe3234033bc665bed4f1b51f5 to start an election
I20240503 14:13:43.658043 9665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95"
dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
from {username='slave'} at 127.0.186.2:36909
I20240503 14:13:43.658309 9665 raft_consensus.cc:482] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:43.658442 9665 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.661329 9665 raft_consensus.cc:504] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:43.662683 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:43.662904 6931 raft_consensus.cc:3032] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:43.662987 6931 raft_consensus.cc:729] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:43.663245 6931 consensus_queue.cc:260] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:43.663687 6931 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.666608 6931 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 2.
I20240503 14:13:43.668292 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:43.668493 7187 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.668489 9665 leader_election.cc:288] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 2 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:43.671191 7187 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 2.
I20240503 14:13:43.672000 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:43.672240 9797 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:43.675004 9797 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 2.
I20240503 14:13:43.676934 12728 raft_consensus.cc:1050] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:43.677724 12728 consensus_queue.cc:237] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:43.678421 9600 leader_election.cc:302] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 7186ed8510424fd597173eaf10c57b12, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:43.678804 13074 raft_consensus.cc:2781] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:43.678964 13074 raft_consensus.cc:686] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:43.679257 13074 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:43.679916 9665 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:43.680600 12411 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:43.696682 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:43.706480 13178 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:43.707427 13189 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:43.713395 13179 tablet.cc:1722] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:43.734153 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=bfbd326ff9fe4a4f90e5601716217d82] alter complete (version 3)
I20240503 14:13:43.744999 9929 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:43.749274 9533 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:43.749707 12961 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20240503 14:13:43.750370 12976 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:43.752353 7187 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:43.753336 12961 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:43.764257 6931 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:43.764991 12411 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:43.765344 6659 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 1 to 2, leader changed from 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 2 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:43.769227 12976 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:43.782737 9929 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:43.782933 9533 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:43.789781 12858 consensus_queue.cc:1035] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.795426 6651 catalog_manager.cc:5484] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:43.810621 7190 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:43.817718 13168 consensus_queue.cc:1035] T 27b495c1b7df4f4391820da60484f881 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:43.827275 7059 consensus_queue.cc:237] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:43.835403 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:43.838614 6652 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:43.838891 6652 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:43.846994 6652 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 325917993a6c4453a45f733957a683a8
I20240503 14:13:43.848145 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet 325917993a6c4453a45f733957a683a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:43 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:43.848153 9512 tablet_service.cc:1514] Processing DeleteTablet for tablet 325917993a6c4453a45f733957a683a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:43 UTC) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:43.848443 13185 tablet_replica.cc:330] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:43.848651 13185 raft_consensus.cc:2229] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:43.848995 13185 raft_consensus.cc:2258] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.849920 13242 tablet_replica.cc:330] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:43.850080 13242 raft_consensus.cc:2229] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:43.850303 13242 raft_consensus.cc:2258] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.851819 13242 ts_tablet_manager.cc:1875] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:43.852669 13185 ts_tablet_manager.cc:1875] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:43.856907 13242 ts_tablet_manager.cc:1888] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:43.857043 13242 log.cc:1196] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/325917993a6c4453a45f733957a683a8
I20240503 14:13:43.857470 13242 ts_tablet_manager.cc:1909] T 325917993a6c4453a45f733957a683a8 P 5b7e36bf354f480c90859cb8867b59cf: Deleting consensus metadata
I20240503 14:13:43.859048 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 325917993a6c4453a45f733957a683a8 (table rebalancer_dynamic_table_new_name [id=bfbd326ff9fe4a4f90e5601716217d82]) successfully deleted
I20240503 14:13:43.859297 6798 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:43.859338 9665 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:43.859912 12813 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:43.860424 12813 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:43.864861 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 325917993a6c4453a45f733957a683a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:43 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:43.878336 7190 consensus_queue.cc:237] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
W20240503 14:13:43.881062 6997 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e0bb82edac2464b9d618c1c80807fb2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:43.882917 13247 tablet_replica.cc:330] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:43.882898 9929 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:43.883414 13247 raft_consensus.cc:2229] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:43.883994 13247 raft_consensus.cc:2258] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.886210 13247 ts_tablet_manager.cc:1875] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:43.896293 9512 tablet_service.cc:1467] Processing CreateTablet for tablet c3f2bd341493482f9519e6dd74cd2f3d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.896531 9910 tablet_service.cc:1467] Processing CreateTablet for tablet c3f2bd341493482f9519e6dd74cd2f3d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.896775 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f2bd341493482f9519e6dd74cd2f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.897114 9910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f2bd341493482f9519e6dd74cd2f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.899729 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 61b62d3aa0bc46c3be67eadc21466ec6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.900166 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b62d3aa0bc46c3be67eadc21466ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.900082 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 61b62d3aa0bc46c3be67eadc21466ec6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.900269 6929 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:43.900599 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b62d3aa0bc46c3be67eadc21466ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.901540 9644 tablet_service.cc:1467] Processing CreateTablet for tablet c3f2bd341493482f9519e6dd74cd2f3d (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.902094 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f2bd341493482f9519e6dd74cd2f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.902076 12873 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:43.904978 12873 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:43.911317 6929 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:43.912588 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 61b62d3aa0bc46c3be67eadc21466ec6 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.913022 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b62d3aa0bc46c3be67eadc21466ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.919668 9513 tablet_service.cc:1467] Processing CreateTablet for tablet d133b4518f1f4decba0ba32cd82e528e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.919814 9779 tablet_service.cc:1467] Processing CreateTablet for tablet d133b4518f1f4decba0ba32cd82e528e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.920106 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d133b4518f1f4decba0ba32cd82e528e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.920373 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d133b4518f1f4decba0ba32cd82e528e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.921927 6651 catalog_manager.cc:5484] T b97ddea52b624c668cb0b28ab0f38196 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 2 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:43.929232 9908 tablet_service.cc:1514] Processing DeleteTablet for tablet b97ddea52b624c668cb0b28ab0f38196 with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:43.930068 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b97ddea52b624c668cb0b28ab0f38196 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:43.941115 9645 tablet_service.cc:1467] Processing CreateTablet for tablet d133b4518f1f4decba0ba32cd82e528e (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:43.941606 9645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d133b4518f1f4decba0ba32cd82e528e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.943794 13249 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:43.952607 6658 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:43.955705 9661 raft_consensus.cc:1261] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:43.958957 13247 ts_tablet_manager.cc:1888] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:43.960036 13250 tablet_replica.cc:330] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:43.960381 13250 raft_consensus.cc:2229] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:43.960606 13250 raft_consensus.cc:2258] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:43.960564 12877 consensus_queue.cc:1035] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:43.961335 13250 ts_tablet_manager.cc:1875] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:43.964208 13251 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:43.966145 13251 tablet_bootstrap.cc:654] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.968115 13249 tablet_bootstrap.cc:654] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:43.965940 9661 raft_consensus.cc:2932] T b97ddea52b624c668cb0b28ab0f38196 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:43.972046 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 34eb5819c4f34147b8dabad7f126a0d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104]), partition=
I20240503 14:13:43.972460 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34eb5819c4f34147b8dabad7f126a0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.973902 13247 log.cc:1196] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/325917993a6c4453a45f733957a683a8
I20240503 14:13:43.974996 13247 ts_tablet_manager.cc:1909] T 325917993a6c4453a45f733957a683a8 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:43.974375 12805 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:43.976374 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 34eb5819c4f34147b8dabad7f126a0d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104]), partition=
I20240503 14:13:43.977404 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 325917993a6c4453a45f733957a683a8 (table rebalancer_dynamic_table_new_name [id=bfbd326ff9fe4a4f90e5601716217d82]) successfully deleted
I20240503 14:13:43.976728 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34eb5819c4f34147b8dabad7f126a0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.984546 13185 ts_tablet_manager.cc:1888] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:43.984689 13185 log.cc:1196] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/325917993a6c4453a45f733957a683a8
I20240503 14:13:43.985498 13185 ts_tablet_manager.cc:1909] T 325917993a6c4453a45f733957a683a8 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:43.986904 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet 325917993a6c4453a45f733957a683a8 (table rebalancer_dynamic_table_new_name [id=bfbd326ff9fe4a4f90e5601716217d82]) successfully deleted
I20240503 14:13:43.989109 9665 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:43.992540 13252 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:43.992980 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 34eb5819c4f34147b8dabad7f126a0d1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104]), partition=
I20240503 14:13:43.993432 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34eb5819c4f34147b8dabad7f126a0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:43.994110 13254 tablet_bootstrap.cc:492] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:43.995559 13248 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:43.997992 13254 tablet_bootstrap.cc:654] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.002686 13255 tablet_bootstrap.cc:492] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:44.004329 13253 tablet_bootstrap.cc:492] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:44.010274 13254 tablet_bootstrap.cc:492] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:44.010995 13254 ts_tablet_manager.cc:1367] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20240503 14:13:44.012619 13253 tablet_bootstrap.cc:654] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.012933 13255 tablet_bootstrap.cc:654] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.014252 13254 raft_consensus.cc:348] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.014561 13254 raft_consensus.cc:374] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.014591 13252 tablet_bootstrap.cc:654] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.014695 13254 raft_consensus.cc:729] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.015023 13254 consensus_queue.cc:260] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.015903 13254 ts_tablet_manager.cc:1398] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:44.017853 13248 tablet_bootstrap.cc:654] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.019821 6652 catalog_manager.cc:5484] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:44.026517 13250 ts_tablet_manager.cc:1888] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:44.026806 13250 log.cc:1196] T b97ddea52b624c668cb0b28ab0f38196 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/b97ddea52b624c668cb0b28ab0f38196
I20240503 14:13:44.027761 13251 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:44.028028 13251 ts_tablet_manager.cc:1367] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.064s user 0.001s sys 0.005s
I20240503 14:13:44.028137 13255 tablet_bootstrap.cc:492] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:44.028398 13255 ts_tablet_manager.cc:1367] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20240503 14:13:44.029052 13251 raft_consensus.cc:348] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.029316 13251 raft_consensus.cc:374] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.029453 13251 raft_consensus.cc:729] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.029531 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet b97ddea52b624c668cb0b28ab0f38196 (table rebalancer_extra_table_30 [id=2d1344a2221244949e59b2403daea42b]) successfully deleted
I20240503 14:13:44.029757 13251 consensus_queue.cc:260] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.030733 13251 ts_tablet_manager.cc:1398] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:44.031077 13251 tablet_bootstrap.cc:492] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:44.030773 13255 raft_consensus.cc:348] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.032069 13255 raft_consensus.cc:374] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.032387 13255 raft_consensus.cc:729] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.032816 13251 tablet_bootstrap.cc:654] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.032990 13255 consensus_queue.cc:260] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.035073 13255 ts_tablet_manager.cc:1398] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 14:13:44.037772 13252 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:44.038036 13252 ts_tablet_manager.cc:1367] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.002s
I20240503 14:13:44.039108 13252 raft_consensus.cc:348] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.041080 13248 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:44.041668 13252 raft_consensus.cc:374] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.041945 13252 raft_consensus.cc:729] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.042338 13252 consensus_queue.cc:260] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.042732 13248 ts_tablet_manager.cc:1367] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.005s
I20240503 14:13:44.045363 13252 ts_tablet_manager.cc:1398] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 14:13:44.046075 13252 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:44.044510 13248 raft_consensus.cc:348] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.048318 13248 raft_consensus.cc:374] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.048426 13248 raft_consensus.cc:729] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.050627 13252 tablet_bootstrap.cc:654] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.048694 13248 consensus_queue.cc:260] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.064235 13248 ts_tablet_manager.cc:1398] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
W20240503 14:13:44.064847 13244 log.cc:924] Time spent T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240503 14:13:44.067253 13251 tablet_bootstrap.cc:492] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:44.067623 13251 ts_tablet_manager.cc:1367] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.008s
I20240503 14:13:44.068975 6798 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:44.077646 13251 raft_consensus.cc:348] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.077898 13251 raft_consensus.cc:374] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.077963 13251 raft_consensus.cc:729] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.078070 12313 raft_consensus.cc:482] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.078182 13251 consensus_queue.cc:260] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.078256 12313 raft_consensus.cc:504] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.079214 12313 leader_election.cc:288] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:44.080034 9533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b62d3aa0bc46c3be67eadc21466ec6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:44.080346 13251 ts_tablet_manager.cc:1398] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
W20240503 14:13:44.081056 9865 leader_election.cc:341] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:44.082182 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b62d3aa0bc46c3be67eadc21466ec6" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
W20240503 14:13:44.084458 9863 leader_election.cc:341] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:44.084681 13269 ts_tablet_manager.cc:902] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:44.084631 9863 leader_election.cc:302] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:44.084899 12313 raft_consensus.cc:2726] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:44.092279 13269 tablet_copy_client.cc:308] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:44.093150 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 0cb6a131c94047649646ad83a0074133 from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:47571)
I20240503 14:13:44.093353 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 0cb6a131c94047649646ad83a0074133 from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:47571: session id = f1a669e28f014fe287932acf56bebc13-0cb6a131c94047649646ad83a0074133
I20240503 14:13:44.096467 9819 tablet_copy_source_session.cc:205] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:44.096856 13249 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:44.097035 13249 ts_tablet_manager.cc:1367] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.153s user 0.005s sys 0.000s
I20240503 14:13:44.098065 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb6a131c94047649646ad83a0074133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:44.097985 13249 raft_consensus.cc:348] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.098210 13249 raft_consensus.cc:374] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.098287 13249 raft_consensus.cc:729] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.098562 13249 consensus_queue.cc:260] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.098807 13253 tablet_bootstrap.cc:492] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:44.098963 13253 ts_tablet_manager.cc:1367] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.096s user 0.014s sys 0.012s
I20240503 14:13:44.100944 13252 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:44.101109 13249 ts_tablet_manager.cc:1398] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:44.101282 13252 ts_tablet_manager.cc:1367] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.007s
I20240503 14:13:44.102250 13252 raft_consensus.cc:348] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.099869 13253 raft_consensus.cc:348] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.102705 13253 raft_consensus.cc:374] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.102874 13253 raft_consensus.cc:729] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.103297 13253 consensus_queue.cc:260] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.103576 13252 raft_consensus.cc:374] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.103803 13252 raft_consensus.cc:729] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.107611 13252 consensus_queue.cc:260] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.107895 13253 ts_tablet_manager.cc:1398] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240503 14:13:44.108510 13252 ts_tablet_manager.cc:1398] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 14:13:44.108980 9929 consensus_queue.cc:237] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:44.110347 13252 tablet_bootstrap.cc:492] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:44.112087 13252 tablet_bootstrap.cc:654] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.113220 13269 tablet_copy_client.cc:783] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:44.113562 13269 tablet_copy_client.cc:647] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:44.069933 13248 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:44.118232 13269 tablet_copy_client.cc:523] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:44.122732 13269 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:44.134631 13248 tablet_bootstrap.cc:654] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
W20240503 14:13:44.143823 9864 consensus_peers.cc:485] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84b58d63957848d7a8b0eae9f60bbbf1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:44.145530 9531 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:44.147408 12959 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:44.152029 13269 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:44.152563 13269 tablet_bootstrap.cc:492] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:44.152506 9797 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:44.153040 13269 ts_tablet_manager.cc:1367] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.013s
I20240503 14:13:44.153457 13158 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:44.153839 13269 raft_consensus.cc:348] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:44.154140 13269 raft_consensus.cc:729] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:44.154474 13269 consensus_queue.cc:260] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:44.154301 12959 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:44.157001 13269 ts_tablet_manager.cc:1398] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 14:13:44.157883 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-0cb6a131c94047649646ad83a0074133 received from {username='slave'} at 127.0.186.1:47571
I20240503 14:13:44.158032 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session f1a669e28f014fe287932acf56bebc13-0cb6a131c94047649646ad83a0074133 on tablet 0cb6a131c94047649646ad83a0074133 with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:44.159050 13236 raft_consensus.cc:482] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.159220 13236 raft_consensus.cc:504] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.159951 13236 leader_election.cc:288] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:44.160979 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b62d3aa0bc46c3be67eadc21466ec6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:44.161271 9928 raft_consensus.cc:2454] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:44.161839 6865 leader_election.cc:302] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:44.165125 9533 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:44.167135 12976 raft_consensus.cc:2781] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:44.167261 12976 raft_consensus.cc:482] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:44.167349 12976 raft_consensus.cc:3037] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.173990 12976 raft_consensus.cc:504] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.174672 12976 leader_election.cc:288] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:44.176016 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b62d3aa0bc46c3be67eadc21466ec6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:44.176079 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b62d3aa0bc46c3be67eadc21466ec6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:44.176308 9928 raft_consensus.cc:3037] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:44.176805 6867 leader_election.cc:341] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:44.177173 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b62d3aa0bc46c3be67eadc21466ec6" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
W20240503 14:13:44.177842 6867 leader_election.cc:341] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:44.182404 9928 raft_consensus.cc:2454] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:44.183508 6865 leader_election.cc:302] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters: 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:44.184604 13237 raft_consensus.cc:2781] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:44.184732 13237 raft_consensus.cc:686] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:44.184980 13237 consensus_queue.cc:237] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.192235 7187 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:44.192955 13158 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:44.197899 6929 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:44.199002 13074 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:44.214036 13074 raft_consensus.cc:482] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.214223 13074 raft_consensus.cc:504] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.214936 13074 leader_election.cc:288] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:44.216267 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:44.216536 9531 raft_consensus.cc:2454] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:44.217263 9601 leader_election.cc:302] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:44.217769 13074 raft_consensus.cc:2781] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:44.217906 13074 raft_consensus.cc:482] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:44.217999 13074 raft_consensus.cc:3037] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.218457 9661 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:44.221256 13074 raft_consensus.cc:504] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.223260 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:44.223620 9930 raft_consensus.cc:3037] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.228292 9930 raft_consensus.cc:2454] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:44.228685 12873 raft_consensus.cc:482] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.228893 12873 raft_consensus.cc:504] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.229672 12873 leader_election.cc:288] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:44.230132 9599 leader_election.cc:302] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:44.230377 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb5819c4f34147b8dabad7f126a0d1" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:44.230628 7059 raft_consensus.cc:2454] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:44.231112 7127 leader_election.cc:302] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:44.231961 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:44.232241 9928 raft_consensus.cc:2362] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1. Re-sending same reply.
I20240503 14:13:44.233023 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb5819c4f34147b8dabad7f126a0d1" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
W20240503 14:13:44.233538 7125 leader_election.cc:341] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:44.235946 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:44.236196 9530 raft_consensus.cc:3037] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.237085 12873 raft_consensus.cc:2781] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:44.237170 12873 raft_consensus.cc:482] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:44.237236 12873 raft_consensus.cc:3037] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.240139 12873 raft_consensus.cc:504] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.241572 7187 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:44.242308 13158 raft_consensus.cc:2781] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:44.243288 9530 raft_consensus.cc:2454] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:44.247370 13288 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:44.241576 13074 leader_election.cc:288] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:44.251673 7190 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:44.252082 6650 catalog_manager.cc:5484] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:44.255925 13158 raft_consensus.cc:686] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:44.255896 13074 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:44.256340 13158 consensus_queue.cc:237] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.259840 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b154f780fc742fa9b0efd609c307f95 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:44.266629 9661 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:44.269254 13282 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:44.274806 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb5819c4f34147b8dabad7f126a0d1" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:44.275046 7059 raft_consensus.cc:3037] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.277937 7059 raft_consensus.cc:2454] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:44.278559 7127 leader_election.cc:302] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:44.278913 12877 raft_consensus.cc:2781] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:44.279031 12877 raft_consensus.cc:686] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:44.279318 12877 consensus_queue.cc:237] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.290164 13248 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:44.290621 6651 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 8 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:44.291517 6651 catalog_manager.cc:5484] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:44.291102 13248 ts_tablet_manager.cc:1367] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.221s user 0.001s sys 0.003s
I20240503 14:13:44.293200 12873 leader_election.cc:288] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:44.294423 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb5819c4f34147b8dabad7f126a0d1" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:44.294687 13248 raft_consensus.cc:348] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.294917 13248 raft_consensus.cc:374] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.294996 13248 raft_consensus.cc:729] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
W20240503 14:13:44.296628 7125 leader_election.cc:341] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:44.299216 13248 consensus_queue.cc:260] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.301369 13248 ts_tablet_manager.cc:1398] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 14:13:44.301682 13248 tablet_bootstrap.cc:492] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:44.306592 13248 tablet_bootstrap.cc:654] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:44.313720 6910 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b154f780fc742fa9b0efd609c307f95 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:44.328940 13248 tablet_bootstrap.cc:492] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:44.332028 13248 ts_tablet_manager.cc:1367] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 14:13:44.334908 9661 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:44.335974 6650 catalog_manager.cc:5484] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:44.347440 13158 raft_consensus.cc:482] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.347707 13158 raft_consensus.cc:504] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.350522 13248 raft_consensus.cc:348] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.350976 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133b4518f1f4decba0ba32cd82e528e" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:44.351097 13248 raft_consensus.cc:374] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.351193 13248 raft_consensus.cc:729] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.351277 9796 raft_consensus.cc:2454] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:44.352191 13248 consensus_queue.cc:260] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.352309 9600 leader_election.cc:302] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:44.352689 13074 raft_consensus.cc:2781] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:44.352854 13074 raft_consensus.cc:482] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:44.352969 13074 raft_consensus.cc:3037] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.353276 13248 ts_tablet_manager.cc:1398] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 14:13:44.356978 13158 leader_election.cc:288] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:44.358738 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133b4518f1f4decba0ba32cd82e528e" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:44.359033 9530 raft_consensus.cc:2454] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 0.
I20240503 14:13:44.362141 13074 raft_consensus.cc:504] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.363085 13074 leader_election.cc:288] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:44.364188 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133b4518f1f4decba0ba32cd82e528e" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:44.364459 9796 raft_consensus.cc:3037] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.366789 9796 raft_consensus.cc:2454] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:44.367250 9600 leader_election.cc:302] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:44.367528 13288 raft_consensus.cc:2781] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:44.367899 13288 raft_consensus.cc:686] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:44.368314 13288 consensus_queue.cc:237] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:44.369705 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133b4518f1f4decba0ba32cd82e528e" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:44.370002 9530 raft_consensus.cc:3037] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.374857 9530 raft_consensus.cc:2454] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 1.
I20240503 14:13:44.381335 6659 catalog_manager.cc:5484] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:44.386032 13283 raft_consensus.cc:482] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.386240 13283 raft_consensus.cc:504] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:44.387163 13283 leader_election.cc:288] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:44.388166 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b62d3aa0bc46c3be67eadc21466ec6" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:44.388478 9929 raft_consensus.cc:2379] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5b7e36bf354f480c90859cb8867b59cf in current term 1: Already voted for candidate 7186ed8510424fd597173eaf10c57b12 in this term.
I20240503 14:13:44.390645 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61b62d3aa0bc46c3be67eadc21466ec6" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:44.391088 13252 tablet_bootstrap.cc:492] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:44.391273 13252 ts_tablet_manager.cc:1367] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.281s user 0.038s sys 0.030s
I20240503 14:13:44.391881 9467 leader_election.cc:302] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:44.392308 13252 raft_consensus.cc:348] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.392529 13252 raft_consensus.cc:374] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:44.392606 13293 tablet_replica.cc:330] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:44.392642 13252 raft_consensus.cc:729] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:44.392894 13293 raft_consensus.cc:2229] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:44.392880 13252 consensus_queue.cc:260] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.393479 13252 ts_tablet_manager.cc:1398] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:44.394381 6652 catalog_manager.cc:5484] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:44.394920 13283 raft_consensus.cc:3037] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.398381 13293 pending_rounds.cc:62] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Trying to abort 1 pending ops.
I20240503 14:13:44.400439 13293 pending_rounds.cc:68] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Aborting op as it isn't in flight: id { term: 2 index: 9 } timestamp: 7023598074596155392 op_type: NO_OP noop_request { }
I20240503 14:13:44.401532 13293 raft_consensus.cc:2866] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:13:44.402218 13293 raft_consensus.cc:2258] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:44.403777 13283 raft_consensus.cc:2726] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:44.405400 13293 ts_tablet_manager.cc:1875] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:44.424696 9798 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:44.425597 13288 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:44.427196 9798 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:44.473040 6929 consensus_queue.cc:237] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
W20240503 14:13:44.499359 6865 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b3ddfa17abc4ad2a64d8015724b4ae4. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:44.499475 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:44.505005 7059 raft_consensus.cc:1261] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:44.506086 12976 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:44.510561 9797 raft_consensus.cc:1261] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:44.511629 12976 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:44.514719 12976 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:44.521376 7059 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:44.533999 6659 catalog_manager.cc:5484] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:44.571735 6798 raft_consensus.cc:1203] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:44.598937 9797 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:44.673678 7059 consensus_queue.cc:237] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:44.687368 13237 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 14:13:44.711105 6995 consensus_peers.cc:485] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3160f1a600604f258f76614f13429563. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:44.714350 6798 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:44.715066 13243 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:44.727922 12961 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.771034 9798 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:44.771780 13243 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:44.785549 13243 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:44.788715 9797 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:44.798637 12959 raft_consensus.cc:482] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:44.798841 12959 raft_consensus.cc:504] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } }
I20240503 14:13:44.799846 12959 leader_election.cc:288] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:44.799926 7059 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb5819c4f34147b8dabad7f126a0d1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:44.800254 7059 raft_consensus.cc:2379] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea801601713a4d55aee0de9e3473e0bc in current term 1: Already voted for candidate e1bf5fb7402c46829974e2f74cd88708 in this term.
I20240503 14:13:44.803794 6651 catalog_manager.cc:5484] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
W20240503 14:13:44.810823 7125 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 375484us
I20240503 14:13:44.819689 7059 raft_consensus.cc:1261] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:44.826184 12877 consensus_queue.cc:1035] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.834249 13288 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.836002 9929 raft_consensus.cc:3037] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:44.860505 9929 raft_consensus.cc:1261] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:44.867743 13288 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.902590 13334 ts_tablet_manager.cc:902] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:44.904129 13333 ts_tablet_manager.cc:902] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:44.917703 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34eb5819c4f34147b8dabad7f126a0d1" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
W20240503 14:13:44.932917 7127 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 106817us
I20240503 14:13:44.948485 7190 consensus_queue.cc:237] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:44.950839 13288 consensus_queue.cc:1035] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.960304 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:44.966907 13306 ts_tablet_manager.cc:902] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:44.968997 13293 ts_tablet_manager.cc:1888] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:13:44.972738 13074 consensus_queue.cc:1035] T d133b4518f1f4decba0ba32cd82e528e P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:44.979513 13333 tablet_copy_client.cc:308] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:44.981580 13334 tablet_copy_client.cc:308] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
W20240503 14:13:44.971148 9533 thread.cc:636] tablet-copy [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.100s sys 0.102s
W20240503 14:13:45.002096 9533 thread.cc:602] tablet-copy [worker] (thread pool) Time spent starting thread: real 0.547s user 0.100s sys 0.103s
I20240503 14:13:45.002545 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:59965)
I20240503 14:13:45.002734 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:59965: session id = f1a669e28f014fe287932acf56bebc13-1b3ddfa17abc4ad2a64d8015724b4ae4
I20240503 14:13:45.005049 9533 raft_consensus.cc:1261] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.005662 6951 tablet_copy_source_session.cc:205] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:45.006125 12877 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:45.009245 13334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.002545 13293 log.cc:1196] T 9b154f780fc742fa9b0efd609c307f95 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b154f780fc742fa9b0efd609c307f95
I20240503 14:13:45.013986 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 9b154f780fc742fa9b0efd609c307f95 (table rebalance_test_table_17 [id=e797cf989a264225bdf1eb7fcdf69bbc]) successfully deleted
I20240503 14:13:45.016141 9864 leader_election.cc:302] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:45.018249 12959 raft_consensus.cc:2726] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240503 14:13:45.059247 7127 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 52070us
I20240503 14:13:45.078836 6650 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 34eb5819c4f34147b8dabad7f126a0d1 (table rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104]). Expected version 1 got 0
I20240503 14:13:45.086663 7039 tablet_service.cc:1467] Processing CreateTablet for tablet 92b477db785d47749fabf31ef385e9d2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.087288 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b477db785d47749fabf31ef385e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.093106 13352 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:45.094944 9908 tablet_service.cc:1467] Processing CreateTablet for tablet 92b477db785d47749fabf31ef385e9d2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.095505 9908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b477db785d47749fabf31ef385e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.095136 13352 tablet.cc:1620] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:45.096611 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 933c08460b9745feab9bcb56c27bdbb9 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.097025 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933c08460b9745feab9bcb56c27bdbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.107396 6910 tablet_service.cc:1467] Processing CreateTablet for tablet 933c08460b9745feab9bcb56c27bdbb9 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.107864 6910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933c08460b9745feab9bcb56c27bdbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.112848 6911 tablet_service.cc:1467] Processing CreateTablet for tablet b9a0d6f38293413aba812c473ab0973e (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.113265 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a0d6f38293413aba812c473ab0973e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 14:13:45.113452 7125 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 54255us
I20240503 14:13:45.114331 13304 consensus_queue.cc:1035] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.116132 13355 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240503 14:13:45.116775 7125 consensus_peers.cc:485] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1423a44f18494d7baece06af1d005ea0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:45.116827 9661 raft_consensus.cc:1261] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.117697 13304 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:45.118525 13355 tablet.cc:1620] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:45.118643 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet 84b58d63957848d7a8b0eae9f60bbbf1 from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:48587)
I20240503 14:13:45.118841 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet 84b58d63957848d7a8b0eae9f60bbbf1 from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:48587: session id = e1bf5fb7402c46829974e2f74cd88708-84b58d63957848d7a8b0eae9f60bbbf1
I20240503 14:13:45.124055 9950 tablet_copy_source_session.cc:205] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:45.125901 13333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b58d63957848d7a8b0eae9f60bbbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.132809 13333 tablet_copy_client.cc:783] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:45.133273 13333 tablet_copy_client.cc:647] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:45.134760 9909 tablet_service.cc:1467] Processing CreateTablet for tablet b9a0d6f38293413aba812c473ab0973e (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.135351 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a0d6f38293413aba812c473ab0973e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.139494 7041 tablet_service.cc:1467] Processing CreateTablet for tablet b9a0d6f38293413aba812c473ab0973e (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.139822 13333 tablet_copy_client.cc:523] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:45.143404 13333 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:45.144534 7041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a0d6f38293413aba812c473ab0973e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.145294 6650 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet 34eb5819c4f34147b8dabad7f126a0d1 (table rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104]). Expected version 1 got 0
I20240503 14:13:45.152256 13357 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:45.154366 13357 tablet_bootstrap.cc:654] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.162752 13304 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:45.175712 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 92b477db785d47749fabf31ef385e9d2 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.176223 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b477db785d47749fabf31ef385e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.181792 9661 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:45.182346 6650 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104] alter complete (version 1)
I20240503 14:13:45.184378 13334 tablet_copy_client.cc:783] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:45.186628 12853 raft_consensus.cc:1050] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:45.187625 12853 consensus_queue.cc:237] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:45.189333 13361 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:45.190013 13361 tablet.cc:1620] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:45.190771 13362 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:45.192631 13362 tablet_bootstrap.cc:654] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.193848 13358 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:45.187949 6779 tablet_service.cc:1467] Processing CreateTablet for tablet 933c08460b9745feab9bcb56c27bdbb9 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:45.195274 6779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933c08460b9745feab9bcb56c27bdbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.195554 13358 tablet_bootstrap.cc:654] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.200973 6800 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:45.203930 6650 catalog_manager.cc:5484] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:45.212822 6798 raft_consensus.cc:1261] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:45.213719 13127 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:45.215131 9661 raft_consensus.cc:1261] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:45.215641 13334 tablet_copy_client.cc:647] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:45.215793 13127 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:13:45.217031 6735 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 186581us
I20240503 14:13:45.224599 13334 tablet_copy_client.cc:523] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:45.229245 13357 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:45.229996 13357 ts_tablet_manager.cc:1367] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.000s
I20240503 14:13:45.232323 13373 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:45.235841 6658 catalog_manager.cc:5576] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653) does not have the latest schema for tablet 34eb5819c4f34147b8dabad7f126a0d1 (table rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104]). Expected version 1 got 0
I20240503 14:13:45.236337 13373 tablet_bootstrap.cc:654] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.240010 13357 raft_consensus.cc:348] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.241508 13357 raft_consensus.cc:374] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.243013 13357 raft_consensus.cc:729] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.248252 13127 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:45.250315 13357 consensus_queue.cc:260] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.252418 6797 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:45.262032 9661 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:45.264672 13358 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:45.265348 13334 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:45.265125 13358 ts_tablet_manager.cc:1367] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.003s
I20240503 14:13:45.259666 6650 catalog_manager.cc:5484] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:45.267400 13358 raft_consensus.cc:348] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.268002 13358 raft_consensus.cc:374] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.268150 13358 raft_consensus.cc:729] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.277534 13358 consensus_queue.cc:260] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.283222 13358 ts_tablet_manager.cc:1398] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240503 14:13:45.284126 13358 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:45.288214 13358 tablet_bootstrap.cc:654] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.289019 13372 ts_tablet_manager.cc:902] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:45.293887 13357 ts_tablet_manager.cc:1398] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20240503 14:13:45.295147 13362 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:45.295396 13362 ts_tablet_manager.cc:1367] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.000s
I20240503 14:13:45.296962 13357 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:45.308207 13362 raft_consensus.cc:348] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.308452 13362 raft_consensus.cc:374] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.308527 13362 raft_consensus.cc:729] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.308751 13362 consensus_queue.cc:260] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.309355 13362 ts_tablet_manager.cc:1398] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240503 14:13:45.311324 13373 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:45.315644 9532 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:45.317744 13373 ts_tablet_manager.cc:1367] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.000s
I20240503 14:13:45.319561 13373 raft_consensus.cc:348] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.319808 13373 raft_consensus.cc:374] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.319865 13373 raft_consensus.cc:729] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.323839 13357 tablet_bootstrap.cc:654] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.324939 9529 raft_consensus.cc:1261] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:45.325002 13373 consensus_queue.cc:260] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.328730 13127 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:45.343930 13357 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:45.344398 13357 ts_tablet_manager.cc:1367] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.004s
I20240503 14:13:45.345165 13373 ts_tablet_manager.cc:1398] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.027s user 0.000s sys 0.003s
I20240503 14:13:45.345270 6659 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:45.345594 13373 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:45.350225 13334 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:45.354983 12805 raft_consensus.cc:482] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.355147 12805 raft_consensus.cc:504] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.355998 12805 leader_election.cc:288] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:45.357784 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
W20240503 14:13:45.358979 6995 leader_election.cc:341] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:45.360065 13352 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:45.360599 13352 tablet.cc:1620] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:45.360874 13355 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:45.361289 13355 tablet.cc:1620] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:45.362771 6650 catalog_manager.cc:5576] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337) does not have the latest schema for tablet 34eb5819c4f34147b8dabad7f126a0d1 (table rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104]). Expected version 2 got 1
I20240503 14:13:45.365368 13362 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:45.366398 6930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
W20240503 14:13:45.367116 6995 leader_election.cc:341] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:45.367265 6995 leader_election.cc:302] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c; no voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:45.367789 12805 raft_consensus.cc:2726] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:45.377709 13361 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:45.381000 13361 tablet.cc:1620] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: MemRowSet was empty: no flush needed.
I20240503 14:13:45.382161 13357 raft_consensus.cc:348] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.384150 13333 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:45.384423 13357 raft_consensus.cc:374] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.384572 13333 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:45.384636 13357 raft_consensus.cc:729] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.384881 13333 ts_tablet_manager.cc:1367] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.242s user 0.009s sys 0.008s
I20240503 14:13:45.385135 6658 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=4347a070217d41e4acd3ee2af2b9f104] alter complete (version 2)
I20240503 14:13:45.385843 13333 raft_consensus.cc:348] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:45.386222 13333 raft_consensus.cc:729] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:45.386458 13333 consensus_queue.cc:260] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:45.387681 13333 ts_tablet_manager.cc:1398] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:45.388583 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-84b58d63957848d7a8b0eae9f60bbbf1 received from {username='slave'} at 127.0.186.4:48587
I20240503 14:13:45.388752 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-84b58d63957848d7a8b0eae9f60bbbf1 on tablet 84b58d63957848d7a8b0eae9f60bbbf1 with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:45.389338 13357 consensus_queue.cc:260] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.389942 13306 tablet_copy_client.cc:308] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:45.390661 13373 tablet_bootstrap.cc:654] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.391049 13334 tablet_bootstrap.cc:492] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:45.391361 13334 ts_tablet_manager.cc:1367] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.007s
I20240503 14:13:45.392220 13334 raft_consensus.cc:348] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:45.392515 13334 raft_consensus.cc:729] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:45.392755 13334 consensus_queue.cc:260] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:45.393270 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 6e0bb82edac2464b9d618c1c80807fb2 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:53967)
I20240503 14:13:45.393405 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 6e0bb82edac2464b9d618c1c80807fb2 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:53967: session id = 5b7e36bf354f480c90859cb8867b59cf-6e0bb82edac2464b9d618c1c80807fb2
I20240503 14:13:45.394755 13357 ts_tablet_manager.cc:1398] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20240503 14:13:45.400048 13394 ts_tablet_manager.cc:902] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:45.400732 7081 tablet_copy_source_session.cc:205] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:45.401546 13281 raft_consensus.cc:482] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.401753 13281 raft_consensus.cc:504] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.402580 13281 leader_election.cc:288] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:45.403164 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b477db785d47749fabf31ef385e9d2" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:45.403371 6798 raft_consensus.cc:2454] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:45.403759 9863 leader_election.cc:302] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:45.404019 13281 raft_consensus.cc:2781] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:45.404130 13281 raft_consensus.cc:482] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:45.404209 13281 raft_consensus.cc:3037] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.406936 13281 raft_consensus.cc:504] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.403501 7057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b477db785d47749fabf31ef385e9d2" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:45.407199 13373 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
W20240503 14:13:45.407816 9865 leader_election.cc:341] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:45.407374 13373 ts_tablet_manager.cc:1367] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.062s user 0.000s sys 0.005s
I20240503 14:13:45.409232 7057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b477db785d47749fabf31ef385e9d2" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
W20240503 14:13:45.410197 9865 leader_election.cc:341] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:45.410367 13281 leader_election.cc:288] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:45.414186 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b477db785d47749fabf31ef385e9d2" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:45.414407 6798 raft_consensus.cc:3037] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.414638 13362 tablet_bootstrap.cc:654] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.415971 13358 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:45.416131 13358 ts_tablet_manager.cc:1367] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.000s
I20240503 14:13:45.415941 13306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e0bb82edac2464b9d618c1c80807fb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.416785 13358 raft_consensus.cc:348] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.416971 13358 raft_consensus.cc:374] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.417021 13358 raft_consensus.cc:729] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.417062 6798 raft_consensus.cc:2454] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:45.417554 9863 leader_election.cc:302] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:45.418013 13281 raft_consensus.cc:2781] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:45.418148 13281 raft_consensus.cc:686] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:45.418390 13281 consensus_queue.cc:237] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.420976 13373 raft_consensus.cc:348] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.423835 13373 raft_consensus.cc:374] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.424487 9532 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:45.424929 13372 tablet_copy_client.cc:308] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:45.426492 9661 consensus_queue.cc:237] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:45.425575 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 3160f1a600604f258f76614f13429563 from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:42011)
I20240503 14:13:45.428433 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 3160f1a600604f258f76614f13429563 from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:42011: session id = ea801601713a4d55aee0de9e3473e0bc-3160f1a600604f258f76614f13429563
I20240503 14:13:45.423913 13334 ts_tablet_manager.cc:1398] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20240503 14:13:45.431008 13306 tablet_copy_client.cc:783] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:45.431402 7081 tablet_copy_source_session.cc:205] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:45.431347 13373 raft_consensus.cc:729] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.431936 13373 consensus_queue.cc:260] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.432121 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-1b3ddfa17abc4ad2a64d8015724b4ae4 received from {username='slave'} at 127.0.186.1:59965
I20240503 14:13:45.432224 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session f1a669e28f014fe287932acf56bebc13-1b3ddfa17abc4ad2a64d8015724b4ae4 on tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:45.417241 13358 consensus_queue.cc:260] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.433480 13372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3160f1a600604f258f76614f13429563. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.438318 13358 ts_tablet_manager.cc:1398] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 14:13:45.446168 13358 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:45.446393 13236 raft_consensus.cc:482] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.446552 13236 raft_consensus.cc:504] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.447108 13373 ts_tablet_manager.cc:1398] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.039s user 0.000s sys 0.002s
I20240503 14:13:45.447223 13236 leader_election.cc:288] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:45.447798 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a0d6f38293413aba812c473ab0973e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:45.448050 9929 raft_consensus.cc:2454] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:45.448459 6865 leader_election.cc:302] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:45.449462 7057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a0d6f38293413aba812c473ab0973e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:45.449684 7057 raft_consensus.cc:2454] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:45.449910 13358 tablet_bootstrap.cc:654] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.450446 13236 raft_consensus.cc:2781] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:45.450521 13236 raft_consensus.cc:482] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:45.450578 13236 raft_consensus.cc:3037] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.453271 13236 raft_consensus.cc:504] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.455396 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a0d6f38293413aba812c473ab0973e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:45.455629 9930 raft_consensus.cc:3037] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.458557 13306 tablet_copy_client.cc:647] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:45.459537 13372 tablet_copy_client.cc:783] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:45.459734 13362 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12: No bootstrap required, opened a new log
I20240503 14:13:45.459870 13362 ts_tablet_manager.cc:1367] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.095s user 0.005s sys 0.000s
I20240503 14:13:45.459889 13372 tablet_copy_client.cc:647] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:45.461517 9930 raft_consensus.cc:2454] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:45.461762 13362 raft_consensus.cc:348] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.462028 13362 raft_consensus.cc:374] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.462141 13362 raft_consensus.cc:729] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: FOLLOWER
W20240503 14:13:45.462301 9601 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1461e0716aee4f74acefc639cf79eaf0. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:45.462330 6865 leader_election.cc:302] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:45.462584 13237 raft_consensus.cc:2781] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:45.462735 13237 raft_consensus.cc:686] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:45.462934 13237 consensus_queue.cc:237] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:45.463331 9531 raft_consensus.cc:1261] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.463913 13074 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:45.465109 13306 tablet_copy_client.cc:523] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:45.466220 13372 tablet_copy_client.cc:523] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:45.466173 7057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9a0d6f38293413aba812c473ab0973e" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:45.466382 7057 raft_consensus.cc:3037] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.469514 7057 raft_consensus.cc:2454] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:45.472151 13236 leader_election.cc:288] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:45.462576 13362 consensus_queue.cc:260] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.473038 9929 raft_consensus.cc:1261] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.474429 13074 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:45.477373 13358 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:45.474011 13362 ts_tablet_manager.cc:1398] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:13:45.480579 13372 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:45.481906 13358 ts_tablet_manager.cc:1367] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20240503 14:13:45.483083 13358 raft_consensus.cc:348] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.483426 13306 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:45.489130 13358 raft_consensus.cc:374] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.489285 13358 raft_consensus.cc:729] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.489549 13358 consensus_queue.cc:260] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.491304 6650 catalog_manager.cc:5484] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:45.494729 13288 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:45.497460 13358 ts_tablet_manager.cc:1398] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 14:13:45.522413 9930 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:45.460172 6659 catalog_manager.cc:5484] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:45.545771 6659 catalog_manager.cc:5484] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:45.551993 9528 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:45.558373 13394 tablet_copy_client.cc:308] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:45.559098 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 1423a44f18494d7baece06af1d005ea0 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:43871)
I20240503 14:13:45.559257 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 1423a44f18494d7baece06af1d005ea0 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:43871: session id = 7186ed8510424fd597173eaf10c57b12-1423a44f18494d7baece06af1d005ea0
I20240503 14:13:45.562538 7211 tablet_copy_source_session.cc:205] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:45.573489 13372 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:45.574216 13372 tablet_bootstrap.cc:492] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:45.574585 13372 ts_tablet_manager.cc:1367] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.094s user 0.046s sys 0.000s
I20240503 14:13:45.575702 13372 raft_consensus.cc:348] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:45.575992 13372 raft_consensus.cc:729] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:45.576232 13372 consensus_queue.cc:260] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:45.583376 13372 ts_tablet_manager.cc:1398] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240503 14:13:45.584077 13394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1423a44f18494d7baece06af1d005ea0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.588095 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-3160f1a600604f258f76614f13429563 received from {username='slave'} at 127.0.186.8:42011
I20240503 14:13:45.588219 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-3160f1a600604f258f76614f13429563 on tablet 3160f1a600604f258f76614f13429563 with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:45.591200 13394 tablet_copy_client.cc:783] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:45.591542 13394 tablet_copy_client.cc:647] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:45.593279 13394 tablet_copy_client.cc:523] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:45.600955 13394 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:45.614996 13394 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:45.616916 13394 tablet_bootstrap.cc:492] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:45.617245 13394 ts_tablet_manager.cc:1367] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.002s
I20240503 14:13:45.618000 13394 raft_consensus.cc:348] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:45.618371 13394 raft_consensus.cc:729] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:45.618626 13394 consensus_queue.cc:260] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:45.619488 13394 ts_tablet_manager.cc:1398] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:45.619853 9798 consensus_queue.cc:237] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:45.622706 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-1423a44f18494d7baece06af1d005ea0 received from {username='slave'} at 127.0.186.2:43871
I20240503 14:13:45.622862 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-1423a44f18494d7baece06af1d005ea0 on tablet 1423a44f18494d7baece06af1d005ea0 with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:45.627995 7190 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:45.628012 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef4166613a824863b6c006dafb3783d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:45.628448 9528 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.628625 12853 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:45.629129 13127 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:45.641389 6659 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:45.652161 13352 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:45.656406 13355 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:45.659303 13361 tablet.cc:1722] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:45.674723 6652 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=4347a070217d41e4acd3ee2af2b9f104] alter complete (version 3)
I20240503 14:13:45.686223 13127 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:45.690987 9528 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:45.699491 7187 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:45.700340 6650 catalog_manager.cc:5484] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:45.726861 7190 raft_consensus.cc:1203] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:45.735714 13306 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:45.736354 13306 tablet_bootstrap.cc:492] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:45.736742 13306 ts_tablet_manager.cc:1367] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.255s user 0.065s sys 0.006s
I20240503 14:13:45.737844 13306 raft_consensus.cc:348] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:45.738178 13306 raft_consensus.cc:729] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:45.738440 13306 consensus_queue.cc:260] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:45.739099 13306 ts_tablet_manager.cc:1398] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:45.740936 13237 raft_consensus.cc:482] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.741099 13237 raft_consensus.cc:504] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.741768 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-6e0bb82edac2464b9d618c1c80807fb2 received from {username='slave'} at 127.0.186.5:53967
I20240503 14:13:45.741824 13237 leader_election.cc:288] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:45.741883 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-6e0bb82edac2464b9d618c1c80807fb2 on tablet 6e0bb82edac2464b9d618c1c80807fb2 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:45.742551 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:45.742630 7057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:45.742769 6798 raft_consensus.cc:2454] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:45.742835 7057 raft_consensus.cc:2454] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 0.
I20240503 14:13:45.743310 6865 leader_election.cc:302] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:45.746752 13237 raft_consensus.cc:2781] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:45.746850 13237 raft_consensus.cc:482] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:45.746908 13237 raft_consensus.cc:3037] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.760344 9929 raft_consensus.cc:1203] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:45.764230 13237 raft_consensus.cc:504] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.767196 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:45.767395 6798 raft_consensus.cc:3037] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.770861 6798 raft_consensus.cc:2454] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:45.774242 6865 leader_election.cc:302] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:45.774891 13236 raft_consensus.cc:2781] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:45.774995 13236 raft_consensus.cc:686] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:45.775213 13236 consensus_queue.cc:237] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } }
I20240503 14:13:45.777041 6659 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:45.777240 6659 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:45.783671 6659 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet 34eb5819c4f34147b8dabad7f126a0d1
I20240503 14:13:45.784390 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb5819c4f34147b8dabad7f126a0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:45.785207 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb5819c4f34147b8dabad7f126a0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:45.785764 7171 tablet_service.cc:1514] Processing DeleteTablet for tablet 34eb5819c4f34147b8dabad7f126a0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:45 UTC) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:45.795063 13429 tablet_replica.cc:330] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:45.796016 13428 tablet_replica.cc:330] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:45.796401 13429 raft_consensus.cc:2229] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:45.796941 13428 raft_consensus.cc:2229] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:45.796944 6659 catalog_manager.cc:5484] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:45.797485 13429 raft_consensus.cc:2258] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.797757 13428 raft_consensus.cc:2258] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.798624 13429 ts_tablet_manager.cc:1875] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.801271 13428 ts_tablet_manager.cc:1875] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.807379 13237 leader_election.cc:288] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 1 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:45.808462 7057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:45.808671 7057 raft_consensus.cc:3037] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.811539 7057 raft_consensus.cc:2454] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:45.823132 13429 ts_tablet_manager.cc:1888] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.826752 13427 tablet_replica.cc:330] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:45.827054 13427 raft_consensus.cc:2229] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:45.827270 13427 raft_consensus.cc:2258] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:45.831192 13428 ts_tablet_manager.cc:1888] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.833070 13427 ts_tablet_manager.cc:1875] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:45.838450 6930 raft_consensus.cc:1203] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:45.838955 13429 log.cc:1196] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/34eb5819c4f34147b8dabad7f126a0d1
I20240503 14:13:45.839448 13429 ts_tablet_manager.cc:1909] T 34eb5819c4f34147b8dabad7f126a0d1 P e1bf5fb7402c46829974e2f74cd88708: Deleting consensus metadata
I20240503 14:13:45.840639 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet 34eb5819c4f34147b8dabad7f126a0d1 (table rebalancer_dynamic_table_new_name [id=4347a070217d41e4acd3ee2af2b9f104]) successfully deleted
I20240503 14:13:45.847739 13428 log.cc:1196] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/34eb5819c4f34147b8dabad7f126a0d1
I20240503 14:13:45.848526 13428 ts_tablet_manager.cc:1909] T 34eb5819c4f34147b8dabad7f126a0d1 P ea801601713a4d55aee0de9e3473e0bc: Deleting consensus metadata
I20240503 14:13:45.854607 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 34eb5819c4f34147b8dabad7f126a0d1 (table rebalancer_dynamic_table_new_name [id=4347a070217d41e4acd3ee2af2b9f104]) successfully deleted
I20240503 14:13:45.859246 13427 ts_tablet_manager.cc:1888] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:45.859472 13427 log.cc:1196] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/34eb5819c4f34147b8dabad7f126a0d1
I20240503 14:13:45.860023 13427 ts_tablet_manager.cc:1909] T 34eb5819c4f34147b8dabad7f126a0d1 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:45.861105 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 34eb5819c4f34147b8dabad7f126a0d1 (table rebalancer_dynamic_table_new_name [id=4347a070217d41e4acd3ee2af2b9f104]) successfully deleted
I20240503 14:13:45.870544 13320 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.876259 7059 consensus_queue.cc:237] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:45.883167 9798 raft_consensus.cc:1261] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.887030 12805 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:45.889194 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
W20240503 14:13:45.890954 6995 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c59daa2d12f446899c7bb9970e1694da. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:45.892971 9665 raft_consensus.cc:1261] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:45.899155 7039 tablet_service.cc:1467] Processing CreateTablet for tablet d75f97e8b21145989a1a0012bd2e0200 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]), partition=
I20240503 14:13:45.899552 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75f97e8b21145989a1a0012bd2e0200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.900180 6780 tablet_service.cc:1467] Processing CreateTablet for tablet d75f97e8b21145989a1a0012bd2e0200 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]), partition=
I20240503 14:13:45.900502 6780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75f97e8b21145989a1a0012bd2e0200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.903328 13236 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.904377 9644 tablet_service.cc:1467] Processing CreateTablet for tablet d75f97e8b21145989a1a0012bd2e0200 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]), partition=
I20240503 14:13:45.904413 13243 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:45.904772 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75f97e8b21145989a1a0012bd2e0200. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:45.906119 13358 tablet_bootstrap.cc:492] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:45.907465 13358 tablet_bootstrap.cc:654] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.918696 13373 tablet_bootstrap.cc:492] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:45.920709 13373 tablet_bootstrap.cc:654] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.922696 13358 tablet_bootstrap.cc:492] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:45.922945 13358 ts_tablet_manager.cc:1367] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.006s
I20240503 14:13:45.924135 13358 raft_consensus.cc:348] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.924536 13358 raft_consensus.cc:374] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.924650 13358 raft_consensus.cc:729] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.925036 13358 consensus_queue.cc:260] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.925877 13358 ts_tablet_manager.cc:1398] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:45.927994 13373 tablet_bootstrap.cc:492] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: No bootstrap required, opened a new log
I20240503 14:13:45.928190 13373 ts_tablet_manager.cc:1367] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240503 14:13:45.929102 13373 raft_consensus.cc:348] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.929402 13373 raft_consensus.cc:374] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:45.929510 13373 raft_consensus.cc:729] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: FOLLOWER
I20240503 14:13:45.929759 13373 consensus_queue.cc:260] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.930380 13373 ts_tablet_manager.cc:1398] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:45.931411 13439 tablet_bootstrap.cc:492] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:45.934365 13439 tablet_bootstrap.cc:654] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:45.937959 13439 tablet_bootstrap.cc:492] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:45.938225 13439 ts_tablet_manager.cc:1367] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 14:13:45.939117 13282 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.943462 13434 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:45.947404 9798 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:45.951717 6652 catalog_manager.cc:5484] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:45.957046 7059 raft_consensus.cc:3037] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:45.967061 6798 raft_consensus.cc:1203] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:45.982602 13434 raft_consensus.cc:482] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:45.982833 13434 raft_consensus.cc:504] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:45.983911 13434 leader_election.cc:288] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:45.988291 13281 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:45.992528 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b477db785d47749fabf31ef385e9d2" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:45.994925 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92b477db785d47749fabf31ef385e9d2" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:45.995563 6995 leader_election.cc:302] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c; no voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:45.983455 9798 consensus_queue.cc:237] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:46.001899 6798 raft_consensus.cc:1261] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:46.002815 13127 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:46.007221 13127 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.012570 6798 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.015555 9528 raft_consensus.cc:1203] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:46.027091 9528 raft_consensus.cc:1261] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:46.027871 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0cb6a131c94047649646ad83a0074133 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:46.029166 12853 consensus_queue.cc:1035] T 0cb6a131c94047649646ad83a0074133 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:46.035557 6652 catalog_manager.cc:5484] T 0cb6a131c94047649646ad83a0074133 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.040710 9528 raft_consensus.cc:2932] T 0cb6a131c94047649646ad83a0074133 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.062897 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cb6a131c94047649646ad83a0074133 with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:46.061662 13439 raft_consensus.cc:348] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:46.069186 13439 raft_consensus.cc:374] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.072824 13434 raft_consensus.cc:2726] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:46.077196 13439 raft_consensus.cc:729] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.077569 13439 consensus_queue.cc:260] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:46.078650 6797 consensus_queue.cc:237] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:46.080787 13394 ts_tablet_manager.cc:902] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:46.081446 13439 ts_tablet_manager.cc:1398] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.140s user 0.001s sys 0.001s
I20240503 14:13:46.090269 9798 raft_consensus.cc:1261] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:46.090498 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8477be3143f64d83a4c68f260a66efbf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:46.090847 6930 raft_consensus.cc:1261] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:46.090888 13165 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:46.091468 13353 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:46.092923 13451 ts_tablet_manager.cc:902] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:46.095979 13353 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:46.103400 6930 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:46.107056 13454 tablet_replica.cc:330] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:46.107308 13454 raft_consensus.cc:2229] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:46.107357 13434 raft_consensus.cc:482] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:46.107558 13454 raft_consensus.cc:2258] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:46.107564 13434 raft_consensus.cc:504] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:46.108284 13434 leader_election.cc:288] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:46.108906 6797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75f97e8b21145989a1a0012bd2e0200" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:46.109109 6797 raft_consensus.cc:2454] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:46.109674 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75f97e8b21145989a1a0012bd2e0200" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:46.109629 6659 catalog_manager.cc:5484] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:46.109890 9665 raft_consensus.cc:2454] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 0.
I20240503 14:13:46.110201 6995 leader_election.cc:302] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:46.110477 13243 raft_consensus.cc:2781] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:46.110555 13243 raft_consensus.cc:482] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:46.110613 13243 raft_consensus.cc:3037] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.113135 13243 raft_consensus.cc:504] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:46.113735 13243 leader_election.cc:288] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:46.113894 9663 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:46.115844 6797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75f97e8b21145989a1a0012bd2e0200" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:46.116071 6797 raft_consensus.cc:3037] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.119612 6797 raft_consensus.cc:2454] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:46.119632 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75f97e8b21145989a1a0012bd2e0200" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:46.119877 9665 raft_consensus.cc:3037] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.120247 6995 leader_election.cc:302] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:46.120527 13243 raft_consensus.cc:2781] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:46.120644 13243 raft_consensus.cc:686] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:46.120898 13243 consensus_queue.cc:237] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:46.123176 9665 raft_consensus.cc:2454] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:46.135088 13394 tablet_copy_client.cc:308] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:46.136051 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet ef4166613a824863b6c006dafb3783d3 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:50341)
I20240503 14:13:46.136255 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet ef4166613a824863b6c006dafb3783d3 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:50341: session id = 7186ed8510424fd597173eaf10c57b12-ef4166613a824863b6c006dafb3783d3
I20240503 14:13:46.140061 9819 tablet_copy_source_session.cc:205] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:46.140883 13454 ts_tablet_manager.cc:1875] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:46.141649 13394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4166613a824863b6c006dafb3783d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.146569 9797 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:46.149524 13454 ts_tablet_manager.cc:1888] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:46.149739 13454 log.cc:1196] T 0cb6a131c94047649646ad83a0074133 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/0cb6a131c94047649646ad83a0074133
I20240503 14:13:46.150499 13394 tablet_copy_client.cc:783] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:46.150862 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 0cb6a131c94047649646ad83a0074133 (table rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]) successfully deleted
I20240503 14:13:46.150617 6650 catalog_manager.cc:5484] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 0 to 1, leader changed from <none> to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:46.150815 13394 tablet_copy_client.cc:647] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:46.164683 13394 tablet_copy_client.cc:523] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:46.168279 13451 tablet_copy_client.cc:308] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.6:33971
I20240503 14:13:46.171432 13394 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:46.172585 9685 tablet_copy_service.cc:140] P f03dc37fe3234033bc665bed4f1b51f5: Received BeginTabletCopySession request for tablet 1461e0716aee4f74acefc639cf79eaf0 from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:35985)
I20240503 14:13:46.172801 9685 tablet_copy_service.cc:161] P f03dc37fe3234033bc665bed4f1b51f5: Beginning new tablet copy session on tablet 1461e0716aee4f74acefc639cf79eaf0 from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:35985: session id = 09723f236d8044538d920a37b5d9aa8c-1461e0716aee4f74acefc639cf79eaf0
I20240503 14:13:46.176728 9685 tablet_copy_source_session.cc:205] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:46.179122 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1461e0716aee4f74acefc639cf79eaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.187294 13451 tablet_copy_client.cc:783] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:46.187680 13451 tablet_copy_client.cc:647] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:46.192006 13451 tablet_copy_client.cc:523] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:46.196604 13451 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:46.204061 13320 consensus_queue.cc:1035] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.204242 13394 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:46.204736 13394 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:46.205078 13394 ts_tablet_manager.cc:1367] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.004s
I20240503 14:13:46.205976 13394 raft_consensus.cc:348] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:46.206193 13394 raft_consensus.cc:729] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:46.206354 13394 consensus_queue.cc:260] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:46.206897 13394 ts_tablet_manager.cc:1398] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:46.207705 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-ef4166613a824863b6c006dafb3783d3 received from {username='slave'} at 127.0.186.2:50341
I20240503 14:13:46.207866 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-ef4166613a824863b6c006dafb3783d3 on tablet ef4166613a824863b6c006dafb3783d3 with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:46.218101 13451 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:46.218585 13451 tablet_bootstrap.cc:492] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:46.218843 13451 ts_tablet_manager.cc:1367] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.005s
I20240503 14:13:46.219528 13451 raft_consensus.cc:348] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:46.219784 13451 raft_consensus.cc:729] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:46.219965 13451 consensus_queue.cc:260] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:46.220465 13451 ts_tablet_manager.cc:1398] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:46.221238 9685 tablet_copy_service.cc:342] P f03dc37fe3234033bc665bed4f1b51f5: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-1461e0716aee4f74acefc639cf79eaf0 received from {username='slave'} at 127.0.186.3:35985
I20240503 14:13:46.221407 9685 tablet_copy_service.cc:433] P f03dc37fe3234033bc665bed4f1b51f5: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-1461e0716aee4f74acefc639cf79eaf0 on tablet 1461e0716aee4f74acefc639cf79eaf0 with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:46.224579 13237 consensus_queue.cc:1035] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.256979 6650 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:46.264398 9665 raft_consensus.cc:1261] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:46.266266 13434 consensus_queue.cc:1035] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.278451 12959 raft_consensus.cc:1050] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
I20240503 14:13:46.279836 12959 consensus_queue.cc:237] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:46.285627 6799 raft_consensus.cc:1261] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:46.289918 13243 consensus_queue.cc:1035] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:46.304605 13303 raft_consensus.cc:1050] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:46.305677 13303 consensus_queue.cc:237] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:46.315333 13243 raft_consensus.cc:1050] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:46.316605 13243 consensus_queue.cc:237] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:46.325156 7190 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.326622 13281 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:46.347116 6650 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]). Expected version 1 got 0
I20240503 14:13:46.359781 6659 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]). Expected version 1 got 0
I20240503 14:13:46.360514 9528 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.371788 13281 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:46.380811 6931 raft_consensus.cc:1261] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.381395 9529 raft_consensus.cc:1261] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.382364 9665 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.382903 13305 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:13:46.383575 13305 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:46.385927 13282 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:46.389194 9662 raft_consensus.cc:1261] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.390475 13304 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:46.400739 13236 raft_consensus.cc:1050] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:46.401737 13236 consensus_queue.cc:237] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:46.419014 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:46.444239 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5922e2c6624846b64abf6df41a5601 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.444793 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5922e2c6624846b64abf6df41a5601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.449146 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5922e2c6624846b64abf6df41a5601 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.449656 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5922e2c6624846b64abf6df41a5601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.449858 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 16b01e1bab7949cfb014c446c08434a8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.450348 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b01e1bab7949cfb014c446c08434a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.453816 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 9e5922e2c6624846b64abf6df41a5601 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.454370 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5922e2c6624846b64abf6df41a5601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.454435 13489 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:46.456084 13489 tablet.cc:1620] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:46.456655 13439 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:46.458382 13439 tablet_bootstrap.cc:654] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.462850 13490 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:46.463734 13490 tablet.cc:1620] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:46.468231 13439 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:46.468482 13439 ts_tablet_manager.cc:1367] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240503 14:13:46.468561 13491 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:46.469710 13439 raft_consensus.cc:348] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.470065 13439 raft_consensus.cc:374] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.470170 13439 raft_consensus.cc:729] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.470566 13439 consensus_queue.cc:260] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.471396 13439 ts_tablet_manager.cc:1398] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:46.472307 13495 ts_tablet_manager.cc:902] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:46.476547 6659 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]). Expected version 1 got 0
I20240503 14:13:46.469254 13491 tablet.cc:1620] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:46.482306 6659 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]). Expected version 1 got 0
I20240503 14:13:46.488277 13495 tablet_copy_client.cc:308] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:46.490388 7171 tablet_service.cc:1467] Processing CreateTablet for tablet 16b01e1bab7949cfb014c446c08434a8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.490865 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b01e1bab7949cfb014c446c08434a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.490754 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 69009d7fef744e66800ce1f07b4a62bf (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.490794 9777 tablet_service.cc:1467] Processing CreateTablet for tablet 16b01e1bab7949cfb014c446c08434a8 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.491202 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b01e1bab7949cfb014c446c08434a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.493870 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 69009d7fef744e66800ce1f07b4a62bf (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.494275 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69009d7fef744e66800ce1f07b4a62bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.508270 9909 tablet_service.cc:1467] Processing CreateTablet for tablet 69009d7fef744e66800ce1f07b4a62bf (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:46.508965 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69009d7fef744e66800ce1f07b4a62bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.509097 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69009d7fef744e66800ce1f07b4a62bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.517427 13439 tablet_bootstrap.cc:492] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:46.518368 13282 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:46.520335 13406 raft_consensus.cc:482] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:46.520500 13406 raft_consensus.cc:504] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.521361 13406 leader_election.cc:288] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:46.523572 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5922e2c6624846b64abf6df41a5601" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
W20240503 14:13:46.524502 9600 leader_election.cc:341] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:46.524966 13439 tablet_bootstrap.cc:654] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.527145 13305 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:46.530386 13508 tablet_bootstrap.cc:492] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:46.534582 13508 tablet_bootstrap.cc:654] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.535310 9662 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:46.534518 9531 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:46.541448 6659 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528] alter complete (version 1)
W20240503 14:13:46.543104 9601 leader_election.cc:341] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Not found: Tablet not found: 9e5922e2c6624846b64abf6df41a5601
I20240503 14:13:46.543834 9601 leader_election.cc:302] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f03dc37fe3234033bc665bed4f1b51f5; no voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:46.546852 13508 tablet_bootstrap.cc:492] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:46.546753 6659 catalog_manager.cc:5484] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:46.547219 13508 ts_tablet_manager.cc:1367] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.011s
I20240503 14:13:46.533145 9529 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:46.548691 13508 raft_consensus.cc:348] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.549052 13508 raft_consensus.cc:374] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.549249 13508 raft_consensus.cc:729] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.549566 13508 consensus_queue.cc:260] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.550318 13406 raft_consensus.cc:2726] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:46.551995 13508 ts_tablet_manager.cc:1398] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 14:13:46.560871 6650 catalog_manager.cc:5484] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:46.579577 13439 tablet_bootstrap.cc:492] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:46.579876 13439 ts_tablet_manager.cc:1367] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.005s
I20240503 14:13:46.580756 13439 raft_consensus.cc:348] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.581037 13439 raft_consensus.cc:374] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.581180 13439 raft_consensus.cc:729] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.581465 13439 consensus_queue.cc:260] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.583953 13439 ts_tablet_manager.cc:1398] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 14:13:46.595855 6931 raft_consensus.cc:1203] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:46.595862 6928 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:46.598984 9663 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
W20240503 14:13:46.630264 13483 log.cc:924] Time spent T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708: Append to log took a long time: real 0.260s user 0.000s sys 0.000s
I20240503 14:13:46.630697 7190 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:46.632584 7187 consensus_queue.cc:237] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:46.635529 9528 raft_consensus.cc:1261] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:46.636236 13304 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:46.638636 6931 raft_consensus.cc:1261] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:46.639106 13305 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:46.639428 13303 consensus_queue.cc:1035] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:46.641400 6928 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:46.643443 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1423a44f18494d7baece06af1d005ea0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:46.644928 9528 raft_consensus.cc:2932] T 1423a44f18494d7baece06af1d005ea0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:46.658536 6799 raft_consensus.cc:1261] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.658689 9930 consensus_queue.cc:237] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:46.659276 13320 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:46.661105 7190 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:46.661756 13511 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:46.667177 9531 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:46.655807 6652 catalog_manager.cc:5484] T 1423a44f18494d7baece06af1d005ea0 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:46.668067 13511 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:46.672508 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 1423a44f18494d7baece06af1d005ea0 with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:46.673461 13511 raft_consensus.cc:482] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:46.673628 13511 raft_consensus.cc:504] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.674376 13511 leader_election.cc:288] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:46.674791 13434 raft_consensus.cc:1050] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:46.674757 6658 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:46.675618 13282 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:46.679576 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84b58d63957848d7a8b0eae9f60bbbf1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:46.681002 9531 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:46.681775 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69009d7fef744e66800ce1f07b4a62bf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
W20240503 14:13:46.682371 9864 leader_election.cc:341] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:46.683914 6658 catalog_manager.cc:5484] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:46.686817 6653 catalog_manager.cc:5576] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691) does not have the latest schema for tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]). Expected version 2 got 1
I20240503 14:13:46.688757 9645 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b58d63957848d7a8b0eae9f60bbbf1 with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:46.676723 13434 consensus_queue.cc:237] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:46.698515 9798 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240503 14:13:46.703473 6995 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 224006us
I20240503 14:13:46.709090 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69009d7fef744e66800ce1f07b4a62bf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:46.710114 7187 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:46.712862 7057 raft_consensus.cc:1261] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.716032 13507 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:46.717049 13506 tablet_bootstrap.cc:492] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:46.718686 13506 tablet_bootstrap.cc:654] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.719723 12805 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:46.720182 13237 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:46.709496 9665 raft_consensus.cc:2454] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
W20240503 14:13:46.723649 13498 log.cc:924] Time spent T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20240503 14:13:46.728130 9798 raft_consensus.cc:1261] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:46.728720 13501 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:46.728843 13320 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:46.738152 9863 leader_election.cc:302] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters: e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:46.738672 13511 raft_consensus.cc:2781] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:46.739456 13511 raft_consensus.cc:482] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:46.739718 13511 raft_consensus.cc:3037] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.743808 13511 raft_consensus.cc:504] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.745257 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69009d7fef744e66800ce1f07b4a62bf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
W20240503 14:13:46.746316 9864 leader_election.cc:341] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Illegal state: must be running to vote when last-logged opid is not known
I20240503 14:13:46.748271 13511 leader_election.cc:288] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:46.752522 6798 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:46.772352 7059 raft_consensus.cc:1203] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:46.773142 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet c59daa2d12f446899c7bb9970e1694da from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:49633)
I20240503 14:13:46.773324 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet c59daa2d12f446899c7bb9970e1694da from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:49633: session id = f1a669e28f014fe287932acf56bebc13-c59daa2d12f446899c7bb9970e1694da
I20240503 14:13:46.776613 7081 tablet_copy_source_session.cc:205] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:46.777982 13451 ts_tablet_manager.cc:902] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:46.782647 9929 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:46.786927 13320 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.788082 13522 tablet_replica.cc:330] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:46.788369 13522 raft_consensus.cc:2229] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:46.788668 13522 raft_consensus.cc:2258] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:46.790529 13522 ts_tablet_manager.cc:1875] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:46.793201 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69009d7fef744e66800ce1f07b4a62bf" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:46.793778 9665 raft_consensus.cc:3037] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:46.797399 7059 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.799610 9529 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:46.800510 13518 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:46.801385 13507 tablet_bootstrap.cc:654] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.806030 13501 tablet_bootstrap.cc:654] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.805910 6653 catalog_manager.cc:5484] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.809660 9798 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.796597 9665 raft_consensus.cc:2454] T 69009d7fef744e66800ce1f07b4a62bf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:46.823907 9863 leader_election.cc:302] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5; no voters: e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:46.825086 13282 raft_consensus.cc:2781] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:46.825256 13282 raft_consensus.cc:686] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:46.825501 13282 consensus_queue.cc:237] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.830060 13519 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:46.831622 6653 catalog_manager.cc:5484] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:46.836537 6798 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.839275 6928 consensus_queue.cc:237] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:46.840083 13495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59daa2d12f446899c7bb9970e1694da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:46.846724 6798 raft_consensus.cc:1261] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:46.847210 13237 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:46.848215 9735 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 175579us
I20240503 14:13:46.853449 13501 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:46.853730 13501 ts_tablet_manager.cc:1367] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.206s user 0.004s sys 0.002s
I20240503 14:13:46.854710 13501 raft_consensus.cc:348] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.855075 13501 raft_consensus.cc:374] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.855237 13501 raft_consensus.cc:729] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.855566 13501 consensus_queue.cc:260] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.856395 13501 ts_tablet_manager.cc:1398] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:46.859102 13522 ts_tablet_manager.cc:1888] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:46.859889 13501 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:46.860963 13495 tablet_copy_client.cc:783] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:46.861310 13495 tablet_copy_client.cc:647] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:46.861526 13501 tablet_bootstrap.cc:654] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.863744 13522 log.cc:1196] T 1423a44f18494d7baece06af1d005ea0 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/1423a44f18494d7baece06af1d005ea0
I20240503 14:13:46.865276 13522 tablet_replica.cc:330] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:46.865538 13522 raft_consensus.cc:2229] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:46.865866 13522 raft_consensus.cc:2258] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:46.866578 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 1423a44f18494d7baece06af1d005ea0 (table rebalancer_extra_table_22 [id=f4184716ab774d6fa7d5115e377a4a61]) successfully deleted
I20240503 14:13:46.870268 13522 ts_tablet_manager.cc:1875] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:46.873692 7059 raft_consensus.cc:1261] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:46.876168 13514 consensus_queue.cc:1035] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:46.880355 13514 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.884402 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:46.886651 7059 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.887928 6650 catalog_manager.cc:5484] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:46.893747 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:46.899791 13506 tablet_bootstrap.cc:492] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:46.899991 13506 ts_tablet_manager.cc:1367] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.183s user 0.001s sys 0.005s
I20240503 14:13:46.900943 13506 raft_consensus.cc:348] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.901201 13506 raft_consensus.cc:374] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.901295 13506 raft_consensus.cc:729] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.901574 13506 consensus_queue.cc:260] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:46.902196 13506 ts_tablet_manager.cc:1398] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:46.902562 13506 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:46.902834 13451 tablet_copy_client.cc:308] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:34101
I20240503 14:13:46.902997 13507 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:46.903164 13507 ts_tablet_manager.cc:1367] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.187s user 0.000s sys 0.006s
I20240503 14:13:46.903954 13507 raft_consensus.cc:348] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.904147 13507 raft_consensus.cc:374] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.904227 13507 raft_consensus.cc:729] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.904453 13507 consensus_queue.cc:260] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.905011 13507 ts_tablet_manager.cc:1398] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 14:13:46.905272 13507 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:46.907078 9663 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:46.907361 13506 tablet_bootstrap.cc:654] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.908277 13434 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:46.912830 13507 tablet_bootstrap.cc:654] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:46.920763 6798 raft_consensus.cc:2932] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:46.927765 6799 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:46.929900 13544 tablet_replica.cc:330] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:46.931905 13544 raft_consensus.cc:2229] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:46.932459 13544 raft_consensus.cc:2258] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:46.934434 13544 ts_tablet_manager.cc:1875] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:46.945293 13506 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:46.948212 13506 ts_tablet_manager.cc:1367] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.008s
I20240503 14:13:46.948961 13434 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:46.952308 13489 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:46.952811 13489 tablet.cc:1620] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:46.954367 13506 raft_consensus.cc:348] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.955483 13506 raft_consensus.cc:374] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:46.955914 13506 raft_consensus.cc:729] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:46.956393 13506 consensus_queue.cc:260] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:46.957743 13506 ts_tablet_manager.cc:1398] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 14:13:46.975162 6652 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528] alter complete (version 2)
I20240503 14:13:46.981978 6820 tablet_copy_service.cc:140] P f1a669e28f014fe287932acf56bebc13: Received BeginTabletCopySession request for tablet 8477be3143f64d83a4c68f260a66efbf from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:43895)
I20240503 14:13:46.982218 6820 tablet_copy_service.cc:161] P f1a669e28f014fe287932acf56bebc13: Beginning new tablet copy session on tablet 8477be3143f64d83a4c68f260a66efbf from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:43895: session id = 09723f236d8044538d920a37b5d9aa8c-8477be3143f64d83a4c68f260a66efbf
I20240503 14:13:46.987695 6820 tablet_copy_source_session.cc:205] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:46.983881 13544 ts_tablet_manager.cc:1888] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:46.992075 13544 log.cc:1196] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/1b3ddfa17abc4ad2a64d8015724b4ae4
I20240503 14:13:46.992826 13521 raft_consensus.cc:1050] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:46.993883 13521 consensus_queue.cc:237] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
W20240503 14:13:46.999796 9467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 78294.7us
I20240503 14:13:47.008302 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 1b3ddfa17abc4ad2a64d8015724b4ae4 (table rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]) successfully deleted
I20240503 14:13:47.012739 6928 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:47.015234 13507 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:47.015547 13507 ts_tablet_manager.cc:1367] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.011s
I20240503 14:13:47.015555 13127 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:47.023422 13434 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:47.023939 9531 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:47.024744 7187 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:47.024755 13521 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:47.025396 13127 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240503 14:13:47.029704 13451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8477be3143f64d83a4c68f260a66efbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.042091 13495 tablet_copy_client.cc:523] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:47.046952 13495 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:47.055874 13507 raft_consensus.cc:348] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.057870 13522 ts_tablet_manager.cc:1888] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:47.060181 13507 raft_consensus.cc:374] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.060842 13507 raft_consensus.cc:729] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.062332 13507 consensus_queue.cc:260] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.063192 13515 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:47.067750 13507 ts_tablet_manager.cc:1398] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.052s user 0.003s sys 0.000s
I20240503 14:13:47.070677 9798 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:47.077279 13243 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:47.080242 13522 log.cc:1196] T 84b58d63957848d7a8b0eae9f60bbbf1 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/84b58d63957848d7a8b0eae9f60bbbf1
I20240503 14:13:47.079331 9929 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:47.080473 6800 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:47.080495 6799 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:47.081131 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 84b58d63957848d7a8b0eae9f60bbbf1 (table rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]) successfully deleted
I20240503 14:13:47.081264 13451 tablet_copy_client.cc:783] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:47.081485 13451 tablet_copy_client.cc:647] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:47.084867 13560 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:47.085479 13560 tablet.cc:1620] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:47.086207 6650 catalog_manager.cc:5484] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:47.093458 9529 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:47.095295 13521 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:47.097641 9531 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:47.099793 13495 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:47.100241 13495 tablet_bootstrap.cc:492] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:47.101168 13495 ts_tablet_manager.cc:1367] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.007s
I20240503 14:13:47.101452 6931 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:47.101931 13495 raft_consensus.cc:348] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:47.102154 13495 raft_consensus.cc:729] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:47.103921 7190 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:47.105041 13501 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:47.105201 13501 ts_tablet_manager.cc:1367] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.245s user 0.058s sys 0.073s
I20240503 14:13:47.106009 13501 raft_consensus.cc:348] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.106204 13501 raft_consensus.cc:374] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.106269 13501 raft_consensus.cc:729] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.106524 13501 consensus_queue.cc:260] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.107143 13501 ts_tablet_manager.cc:1398] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:47.108378 13074 raft_consensus.cc:1050] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:47.111187 13304 raft_consensus.cc:482] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.111366 13304 raft_consensus.cc:504] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.111263 9665 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:47.112068 13304 leader_election.cc:288] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:47.112215 6659 catalog_manager.cc:5484] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:47.112811 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b01e1bab7949cfb014c446c08434a8" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:47.113059 9528 raft_consensus.cc:2454] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:47.113675 7127 leader_election.cc:302] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:47.114029 13304 raft_consensus.cc:2781] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.114043 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b01e1bab7949cfb014c446c08434a8" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:47.114148 13304 raft_consensus.cc:482] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.114245 13304 raft_consensus.cc:3037] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.114265 9797 raft_consensus.cc:2454] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 0.
I20240503 14:13:47.116791 13074 consensus_queue.cc:237] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:47.117296 13304 raft_consensus.cc:504] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.118373 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b01e1bab7949cfb014c446c08434a8" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:47.118606 9528 raft_consensus.cc:3037] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
W20240503 14:13:47.122573 9601 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.5:36609, user_credentials={real_user=slave}} blocked reactor thread for 43502.3us
I20240503 14:13:47.123097 13521 raft_consensus.cc:482] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.123297 13521 raft_consensus.cc:504] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.124079 13521 leader_election.cc:288] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:47.125044 9663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5922e2c6624846b64abf6df41a5601" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:47.125461 9663 raft_consensus.cc:2454] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:47.126107 9733 leader_election.cc:302] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:47.126504 13521 raft_consensus.cc:2781] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.126628 13521 raft_consensus.cc:482] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.126724 13521 raft_consensus.cc:3037] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.129549 9528 raft_consensus.cc:2454] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:47.129550 13521 raft_consensus.cc:504] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.130010 7127 leader_election.cc:302] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:47.130331 13512 raft_consensus.cc:2781] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.130401 13521 leader_election.cc:288] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:47.130465 13512 raft_consensus.cc:686] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:47.130736 13512 consensus_queue.cc:237] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.131772 13304 leader_election.cc:288] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:47.131747 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5922e2c6624846b64abf6df41a5601" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:47.131954 9532 raft_consensus.cc:3037] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.132385 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5922e2c6624846b64abf6df41a5601" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:47.132601 9662 raft_consensus.cc:3037] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.132885 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16b01e1bab7949cfb014c446c08434a8" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:47.133059 9797 raft_consensus.cc:3037] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.134964 9532 raft_consensus.cc:2454] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:47.135402 9662 raft_consensus.cc:2454] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:47.135689 9735 leader_election.cc:302] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:47.136003 13449 raft_consensus.cc:2781] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.136168 13449 raft_consensus.cc:686] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:47.136474 13449 consensus_queue.cc:237] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } }
I20240503 14:13:47.140424 9797 raft_consensus.cc:2454] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 1.
I20240503 14:13:47.128367 6653 catalog_manager.cc:5484] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:47.145694 13495 consensus_queue.cc:260] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:47.146915 9529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e5922e2c6624846b64abf6df41a5601" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:47.147203 9529 raft_consensus.cc:2362] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1. Re-sending same reply.
I20240503 14:13:47.160346 9798 consensus_queue.cc:237] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:47.163508 6653 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 14:13:47.164225 6658 catalog_manager.cc:5484] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } }
I20240503 14:13:47.164567 9532 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:47.165416 6930 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:47.173065 13449 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:47.177814 13449 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:47.178997 13449 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:47.183877 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef4166613a824863b6c006dafb3783d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:47.186513 9529 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:47.202806 6931 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:47.212849 6659 catalog_manager.cc:5484] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 0 to 1, leader changed from <none> to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:47.212821 6658 catalog_manager.cc:5484] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 4, VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:47.217188 13495 ts_tablet_manager.cc:1398] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.116s user 0.000s sys 0.002s
I20240503 14:13:47.221666 7170 tablet_service.cc:1514] Processing DeleteTablet for tablet ef4166613a824863b6c006dafb3783d3 with delete_type TABLET_DATA_TOMBSTONED (TS e1bf5fb7402c46829974e2f74cd88708 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50984
I20240503 14:13:47.222409 13575 tablet_replica.cc:330] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: stopping tablet replica
I20240503 14:13:47.223075 13575 raft_consensus.cc:2229] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:47.223366 13575 raft_consensus.cc:2258] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.234915 13575 ts_tablet_manager.cc:1875] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:47.250375 13509 consensus_queue.cc:1035] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.254869 13512 raft_consensus.cc:482] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.255131 13512 raft_consensus.cc:504] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.256387 13512 leader_election.cc:288] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:47.260655 6650 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table_new_name [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]). Expected version 3 got 2
I20240503 14:13:47.260903 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69009d7fef744e66800ce1f07b4a62bf" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:47.262710 9665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69009d7fef744e66800ce1f07b4a62bf" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:47.264789 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-c59daa2d12f446899c7bb9970e1694da received from {username='slave'} at 127.0.186.1:49633
I20240503 14:13:47.264937 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session f1a669e28f014fe287932acf56bebc13-c59daa2d12f446899c7bb9970e1694da on tablet c59daa2d12f446899c7bb9970e1694da with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:47.265755 13451 tablet_copy_client.cc:523] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:47.265771 13489 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:47.269610 13451 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:47.270439 13560 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:47.272131 7125 leader_election.cc:302] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:47.272609 13512 raft_consensus.cc:2726] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 14:13:47.274609 9929 raft_consensus.cc:1261] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:47.278227 9532 raft_consensus.cc:1261] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:47.285923 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table_new_name [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528] alter complete (version 3)
I20240503 14:13:47.293241 7056 raft_consensus.cc:1261] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:47.296757 13567 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20240503 14:13:47.299813 13575 ts_tablet_manager.cc:1888] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:47.301193 13575 log.cc:1196] T ef4166613a824863b6c006dafb3783d3 P e1bf5fb7402c46829974e2f74cd88708: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-3/wal/wals/ef4166613a824863b6c006dafb3783d3
I20240503 14:13:47.302736 6645 catalog_manager.cc:4830] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): tablet ef4166613a824863b6c006dafb3783d3 (table rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]) successfully deleted
I20240503 14:13:47.302325 13074 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20240503 14:13:47.305002 13566 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:47.306708 13451 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:47.307163 13451 tablet_bootstrap.cc:492] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:47.307435 13451 ts_tablet_manager.cc:1367] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.038s user 0.016s sys 0.001s
I20240503 14:13:47.317328 13579 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:47.318552 13579 tablet.cc:1620] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: MemRowSet was empty: no flush needed.
I20240503 14:13:47.325198 13451 raft_consensus.cc:348] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:47.325529 13451 raft_consensus.cc:729] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:47.325810 13451 consensus_queue.cc:260] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:47.333096 6658 catalog_manager.cc:5576] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101) does not have the latest schema for tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table_new_name [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]). Expected version 3 got 2
I20240503 14:13:47.337478 13451 ts_tablet_manager.cc:1398] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 14:13:47.344707 6820 tablet_copy_service.cc:342] P f1a669e28f014fe287932acf56bebc13: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-8477be3143f64d83a4c68f260a66efbf received from {username='slave'} at 127.0.186.3:43895
I20240503 14:13:47.344852 6820 tablet_copy_service.cc:433] P f1a669e28f014fe287932acf56bebc13: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-8477be3143f64d83a4c68f260a66efbf on tablet 8477be3143f64d83a4c68f260a66efbf with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:47.346271 7187 raft_consensus.cc:3037] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.344198 13579 tablet.cc:1722] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 14:13:47.348930 13567 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.349831 9529 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.354084 9929 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.361810 6658 catalog_manager.cc:5484] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.365694 13511 consensus_queue.cc:1035] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.397333 6658 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:47.397594 6658 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 14:13:47.407431 6658 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: Sending DeleteTablet for 3 replicas of tablet d75f97e8b21145989a1a0012bd2e0200
I20240503 14:13:47.409265 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet d75f97e8b21145989a1a0012bd2e0200 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:47 UTC) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:47.410619 6780 tablet_service.cc:1514] Processing DeleteTablet for tablet d75f97e8b21145989a1a0012bd2e0200 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:47 UTC) from {username='slave'} at 127.0.0.1:54990
I20240503 14:13:47.410305 7056 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.410979 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet d75f97e8b21145989a1a0012bd2e0200 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 14:13:47 UTC) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:47.411268 13522 tablet_replica.cc:330] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:47.411497 13522 raft_consensus.cc:2229] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:47.412665 13522 raft_consensus.cc:2258] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.414251 13522 ts_tablet_manager.cc:1875] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:47.424356 13522 ts_tablet_manager.cc:1888] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:47.424561 13522 log.cc:1196] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/d75f97e8b21145989a1a0012bd2e0200
I20240503 14:13:47.425107 9662 consensus_queue.cc:237] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:47.425601 13591 tablet_replica.cc:330] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:47.425913 13591 raft_consensus.cc:2229] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Raft consensus shutting down.
I20240503 14:13:47.426403 13591 raft_consensus.cc:2258] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.426898 13592 tablet_replica.cc:330] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: stopping tablet replica
I20240503 14:13:47.427124 13592 raft_consensus.cc:2229] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:47.427295 13592 raft_consensus.cc:2258] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.427570 9929 raft_consensus.cc:1261] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:47.428036 13591 ts_tablet_manager.cc:1875] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:47.428254 13592 ts_tablet_manager.cc:1875] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 14:13:47.428216 13406 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:47.429970 7059 raft_consensus.cc:1261] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:47.431828 13567 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:47.432305 13566 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.433739 7056 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.425135 13522 ts_tablet_manager.cc:1909] T d75f97e8b21145989a1a0012bd2e0200 P f03dc37fe3234033bc665bed4f1b51f5: Deleting consensus metadata
I20240503 14:13:47.436618 13591 ts_tablet_manager.cc:1888] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:47.436772 13591 log.cc:1196] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/d75f97e8b21145989a1a0012bd2e0200
I20240503 14:13:47.437584 13591 ts_tablet_manager.cc:1909] T d75f97e8b21145989a1a0012bd2e0200 P 09723f236d8044538d920a37b5d9aa8c: Deleting consensus metadata
I20240503 14:13:47.437409 9930 raft_consensus.cc:2932] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.438885 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table_new_name [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]) successfully deleted
I20240503 14:13:47.441193 6650 catalog_manager.cc:5484] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:47.441982 13592 ts_tablet_manager.cc:1888] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 14:13:47.442152 13592 log.cc:1196] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-0/wal/wals/d75f97e8b21145989a1a0012bd2e0200
I20240503 14:13:47.442250 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table_new_name [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]) successfully deleted
I20240503 14:13:47.442618 13592 ts_tablet_manager.cc:1909] T d75f97e8b21145989a1a0012bd2e0200 P f1a669e28f014fe287932acf56bebc13: Deleting consensus metadata
I20240503 14:13:47.443794 6644 catalog_manager.cc:4830] TS f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): tablet d75f97e8b21145989a1a0012bd2e0200 (table rebalancer_dynamic_table_new_name [id=bd7cbbc5ee9e41a1bc36aa0aeed3b528]) successfully deleted
I20240503 14:13:47.444051 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1461e0716aee4f74acefc639cf79eaf0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:47.446316 9513 tablet_service.cc:1514] Processing DeleteTablet for tablet 1461e0716aee4f74acefc639cf79eaf0 with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:47.452029 13595 tablet_replica.cc:330] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:47.452325 13595 raft_consensus.cc:2229] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:47.452705 13595 raft_consensus.cc:2258] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:47.453933 13595 ts_tablet_manager.cc:1875] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:47.460048 13595 ts_tablet_manager.cc:1888] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:47.460183 13595 log.cc:1196] T 1461e0716aee4f74acefc639cf79eaf0 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/1461e0716aee4f74acefc639cf79eaf0
I20240503 14:13:47.461009 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 1461e0716aee4f74acefc639cf79eaf0 (table rebalancer_extra_table_12 [id=9d8da2d8e59b416282845c6ffdb7597d]) successfully deleted
I20240503 14:13:47.465324 6798 raft_consensus.cc:1203] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:47.479529 7056 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3160f1a600604f258f76614f13429563"
dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39848
I20240503 14:13:47.479915 7056 raft_consensus.cc:595] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:47.509943 6650 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 14:13:47.519215 9513 tablet_service.cc:1467] Processing CreateTablet for tablet e14af3987fbc44e896615d5229d815d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67]), partition=
I20240503 14:13:47.519467 7170 tablet_service.cc:1467] Processing CreateTablet for tablet e14af3987fbc44e896615d5229d815d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67]), partition=
I20240503 14:13:47.519889 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14af3987fbc44e896615d5229d815d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.520133 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14af3987fbc44e896615d5229d815d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.523479 9644 tablet_service.cc:1467] Processing CreateTablet for tablet e14af3987fbc44e896615d5229d815d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67]), partition=
I20240503 14:13:47.523919 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14af3987fbc44e896615d5229d815d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.527145 13507 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:47.527417 13449 consensus_queue.cc:1035] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.529042 13507 tablet_bootstrap.cc:654] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.537194 13507 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:47.537389 13507 ts_tablet_manager.cc:1367] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20240503 14:13:47.538802 13507 raft_consensus.cc:348] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.539168 13507 raft_consensus.cc:374] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.539322 13507 raft_consensus.cc:729] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.539731 13507 consensus_queue.cc:260] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.540424 13507 ts_tablet_manager.cc:1398] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 14:13:47.546356 13627 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:47.548111 13627 tablet_bootstrap.cc:654] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.549496 13628 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:47.562492 13515 raft_consensus.cc:979] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: : Instructing follower f1a669e28f014fe287932acf56bebc13 to start an election
I20240503 14:13:47.562700 13515 raft_consensus.cc:1067] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Signalling peer f1a669e28f014fe287932acf56bebc13 to start an election
I20240503 14:13:47.563650 6798 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3160f1a600604f258f76614f13429563"
dest_uuid: "f1a669e28f014fe287932acf56bebc13"
from {username='slave'} at 127.0.186.3:43895
I20240503 14:13:47.563910 6798 raft_consensus.cc:482] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:47.564060 6798 raft_consensus.cc:3037] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:47.567485 6798 raft_consensus.cc:504] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:47.570379 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3160f1a600604f258f76614f13429563" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:47.570500 9798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3160f1a600604f258f76614f13429563" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:47.570626 7056 raft_consensus.cc:3032] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:47.570704 9798 raft_consensus.cc:3037] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:47.570717 7056 raft_consensus.cc:729] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:47.571014 7056 consensus_queue.cc:260] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:47.571462 7056 raft_consensus.cc:3037] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:47.573930 7056 raft_consensus.cc:2454] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:47.576783 9798 raft_consensus.cc:2454] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:47.577606 6735 leader_election.cc:302] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:47.578032 13568 raft_consensus.cc:2781] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:47.578253 13568 raft_consensus.cc:686] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 2 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:47.578637 13568 consensus_queue.cc:237] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:47.581607 9930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3160f1a600604f258f76614f13429563" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:47.581908 9930 raft_consensus.cc:3037] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:47.586109 6798 leader_election.cc:288] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:47.587222 9930 raft_consensus.cc:2454] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:47.590931 13628 tablet_bootstrap.cc:654] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.594331 6658 catalog_manager.cc:5484] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 2 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:47.608815 13305 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.610049 13127 consensus_queue.cc:1035] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.624576 7056 raft_consensus.cc:1203] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:47.631350 13627 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:47.633172 13627 ts_tablet_manager.cc:1367] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.008s
I20240503 14:13:47.634073 13627 raft_consensus.cc:348] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.634270 13627 raft_consensus.cc:374] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.634338 13627 raft_consensus.cc:729] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.634536 13627 consensus_queue.cc:260] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.635167 13627 ts_tablet_manager.cc:1398] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:47.643626 13628 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:47.643887 13628 ts_tablet_manager.cc:1367] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.095s user 0.005s sys 0.003s
I20240503 14:13:47.647181 13628 raft_consensus.cc:348] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.647645 13628 raft_consensus.cc:374] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.647806 13628 raft_consensus.cc:729] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.648156 13628 consensus_queue.cc:260] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.653332 13628 ts_tablet_manager.cc:1398] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 14:13:47.664826 13304 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.669490 7056 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2"
dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39858
I20240503 14:13:47.669759 7056 raft_consensus.cc:595] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:47.709025 12805 raft_consensus.cc:979] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: : Instructing follower 5b7e36bf354f480c90859cb8867b59cf to start an election
I20240503 14:13:47.709213 13434 raft_consensus.cc:1067] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Signalling peer 5b7e36bf354f480c90859cb8867b59cf to start an election
I20240503 14:13:47.709882 9529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2"
dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
from {username='slave'} at 127.0.186.3:55131
I20240503 14:13:47.710096 9529 raft_consensus.cc:482] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:47.710222 9529 raft_consensus.cc:3037] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:47.720242 13549 raft_consensus.cc:482] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.721989 13549 raft_consensus.cc:504] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.722402 9529 raft_consensus.cc:504] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:47.723006 13549 leader_election.cc:288] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:47.723831 9664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14af3987fbc44e896615d5229d815d7" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:47.723954 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:47.724087 9664 raft_consensus.cc:2454] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:47.724192 9662 raft_consensus.cc:3037] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:47.724182 6798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:47.724363 6798 raft_consensus.cc:3037] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:47.723326 9529 leader_election.cc:288] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Requested vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:47.727108 6798 raft_consensus.cc:2454] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:47.728402 9662 raft_consensus.cc:2454] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:47.730930 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e0bb82edac2464b9d618c1c80807fb2" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:47.731134 7056 raft_consensus.cc:3032] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:47.731251 7056 raft_consensus.cc:729] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:47.732792 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14af3987fbc44e896615d5229d815d7" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:47.733001 7190 raft_consensus.cc:2454] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 0.
I20240503 14:13:47.733646 9467 leader_election.cc:302] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:47.733948 13549 raft_consensus.cc:2781] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.734035 9467 leader_election.cc:302] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:47.734122 13549 raft_consensus.cc:482] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.734210 13549 raft_consensus.cc:3037] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.734319 13380 raft_consensus.cc:2781] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:47.731493 7056 consensus_queue.cc:260] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:47.736114 7056 raft_consensus.cc:3037] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:47.738580 13549 raft_consensus.cc:504] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.738848 7056 raft_consensus.cc:2454] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:47.739353 13549 leader_election.cc:288] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:47.743760 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14af3987fbc44e896615d5229d815d7" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:47.743984 7187 raft_consensus.cc:3037] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.746214 7187 raft_consensus.cc:2454] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:47.748427 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14af3987fbc44e896615d5229d815d7" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:47.748656 9662 raft_consensus.cc:3037] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.749087 9468 leader_election.cc:302] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:47.753480 13549 raft_consensus.cc:2781] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.753650 13549 raft_consensus.cc:686] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:47.753969 13549 consensus_queue.cc:237] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:47.755282 9662 raft_consensus.cc:2454] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:47.770907 13380 raft_consensus.cc:686] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:47.771438 13380 consensus_queue.cc:237] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:47.783840 6658 catalog_manager.cc:5484] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:47.788211 6655 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34184:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 14:13:47.801330 9644 tablet_service.cc:1467] Processing CreateTablet for tablet 616e17b094b74234b3c59bc13e655f95 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.801441 7039 tablet_service.cc:1467] Processing CreateTablet for tablet f47877cdf5f04676861b3dc4708f3920 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.801800 9644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616e17b094b74234b3c59bc13e655f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.801877 7039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47877cdf5f04676861b3dc4708f3920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.802500 9909 tablet_service.cc:1467] Processing CreateTablet for tablet f47877cdf5f04676861b3dc4708f3920 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.802954 9909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47877cdf5f04676861b3dc4708f3920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.803661 9513 tablet_service.cc:1467] Processing CreateTablet for tablet 616e17b094b74234b3c59bc13e655f95 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.803941 7040 tablet_service.cc:1467] Processing CreateTablet for tablet 4d04b4d914bc485fa3f437ad0eb40cb5 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.804119 9513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616e17b094b74234b3c59bc13e655f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.804297 7040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d04b4d914bc485fa3f437ad0eb40cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.810513 9779 tablet_service.cc:1467] Processing CreateTablet for tablet f47877cdf5f04676861b3dc4708f3920 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.810729 13628 tablet_bootstrap.cc:492] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:47.811053 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47877cdf5f04676861b3dc4708f3920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.812601 13628 tablet_bootstrap.cc:654] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.814013 7170 tablet_service.cc:1467] Processing CreateTablet for tablet 4d04b4d914bc485fa3f437ad0eb40cb5 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.814455 7170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d04b4d914bc485fa3f437ad0eb40cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.810513 9777 tablet_service.cc:1467] Processing CreateTablet for tablet 4d04b4d914bc485fa3f437ad0eb40cb5 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.814831 9777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d04b4d914bc485fa3f437ad0eb40cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.810545 9778 tablet_service.cc:1467] Processing CreateTablet for tablet 616e17b094b74234b3c59bc13e655f95 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 14:13:47.820283 9778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 616e17b094b74234b3c59bc13e655f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:47.821053 13627 tablet_bootstrap.cc:492] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:47.822491 13627 tablet_bootstrap.cc:654] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.823160 6655 catalog_manager.cc:5484] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:47.830781 13507 tablet_bootstrap.cc:492] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:47.832598 13507 tablet_bootstrap.cc:654] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.842550 13627 tablet_bootstrap.cc:492] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708: No bootstrap required, opened a new log
I20240503 14:13:47.842886 13627 ts_tablet_manager.cc:1367] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
I20240503 14:13:47.843905 13627 raft_consensus.cc:348] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:47.844184 13627 raft_consensus.cc:374] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.844282 13627 raft_consensus.cc:729] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.844558 13627 consensus_queue.cc:260] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:47.848259 13627 ts_tablet_manager.cc:1398] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 14:13:47.859508 13628 tablet_bootstrap.cc:492] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5: No bootstrap required, opened a new log
I20240503 14:13:47.859813 13628 ts_tablet_manager.cc:1367] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.007s
I20240503 14:13:47.860603 13628 raft_consensus.cc:348] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.860945 13628 raft_consensus.cc:374] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.861066 13628 raft_consensus.cc:729] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.861371 13507 tablet_bootstrap.cc:492] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf: No bootstrap required, opened a new log
I20240503 14:13:47.861630 13507 ts_tablet_manager.cc:1367] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.005s
I20240503 14:13:47.861863 13628 consensus_queue.cc:260] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.863948 13507 raft_consensus.cc:348] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.864248 13507 raft_consensus.cc:374] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.864387 13507 raft_consensus.cc:729] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.864727 13507 consensus_queue.cc:260] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.869704 13507 ts_tablet_manager.cc:1398] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 14:13:47.870890 13652 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:47.872027 13653 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:47.874045 13628 ts_tablet_manager.cc:1398] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 14:13:47.878522 13653 tablet_bootstrap.cc:654] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.885450 13649 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:47.886550 13652 tablet_bootstrap.cc:654] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.887986 13649 tablet_bootstrap.cc:654] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.890105 13653 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:47.890326 13653 ts_tablet_manager.cc:1367] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 14:13:47.890933 13653 raft_consensus.cc:348] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.891103 13653 raft_consensus.cc:374] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.891214 13653 raft_consensus.cc:729] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.891449 13653 consensus_queue.cc:260] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.892299 13653 ts_tablet_manager.cc:1398] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:47.892634 13653 tablet_bootstrap.cc:492] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:47.894419 13653 tablet_bootstrap.cc:654] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.896404 13652 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:47.896670 13652 ts_tablet_manager.cc:1367] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240503 14:13:47.897540 13652 raft_consensus.cc:348] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.897815 13652 raft_consensus.cc:374] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.897949 13652 raft_consensus.cc:729] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.898272 13652 consensus_queue.cc:260] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.900388 13652 ts_tablet_manager.cc:1398] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 14:13:47.900736 13652 tablet_bootstrap.cc:492] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:47.904855 13652 tablet_bootstrap.cc:654] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.904917 13653 tablet_bootstrap.cc:492] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c: No bootstrap required, opened a new log
I20240503 14:13:47.905148 13653 ts_tablet_manager.cc:1367] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240503 14:13:47.905967 13653 raft_consensus.cc:348] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:47.906287 13653 raft_consensus.cc:374] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.906422 13653 raft_consensus.cc:729] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.906674 13653 consensus_queue.cc:260] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:47.908437 13653 ts_tablet_manager.cc:1398] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:47.915365 13649 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc: No bootstrap required, opened a new log
I20240503 14:13:47.915540 13649 ts_tablet_manager.cc:1367] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240503 14:13:47.916425 13649 raft_consensus.cc:348] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.916622 13649 raft_consensus.cc:374] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.916711 13649 raft_consensus.cc:729] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.916922 13649 consensus_queue.cc:260] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.917400 13649 ts_tablet_manager.cc:1398] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:47.918604 13652 tablet_bootstrap.cc:492] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:47.918777 13652 ts_tablet_manager.cc:1367] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240503 14:13:47.919454 13652 raft_consensus.cc:348] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.919839 13652 raft_consensus.cc:374] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.919968 13652 raft_consensus.cc:729] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.920243 13652 consensus_queue.cc:260] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.925352 13652 ts_tablet_manager.cc:1398] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 14:13:47.927263 13652 tablet_bootstrap.cc:492] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:47.929033 13652 tablet_bootstrap.cc:654] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c: Neither blocks nor log segments found. Creating new log.
I20240503 14:13:47.931236 13509 raft_consensus.cc:482] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.931442 13509 raft_consensus.cc:504] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.932271 13509 leader_election.cc:288] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:47.933326 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47877cdf5f04676861b3dc4708f3920" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:47.933516 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47877cdf5f04676861b3dc4708f3920" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:47.933601 9797 raft_consensus.cc:2454] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:47.933749 7056 raft_consensus.cc:2454] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 0.
I20240503 14:13:47.934952 9864 leader_election.cc:302] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:47.936371 13282 raft_consensus.cc:2781] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.936509 13282 raft_consensus.cc:482] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.936640 13282 raft_consensus.cc:3037] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.940969 13282 raft_consensus.cc:504] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.942449 13652 tablet_bootstrap.cc:492] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c: No bootstrap required, opened a new log
I20240503 14:13:47.942606 13652 ts_tablet_manager.cc:1367] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240503 14:13:47.943445 13652 raft_consensus.cc:348] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:47.943673 13652 raft_consensus.cc:374] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 14:13:47.943764 13652 raft_consensus.cc:729] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: FOLLOWER
I20240503 14:13:47.943969 13652 consensus_queue.cc:260] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:47.944545 13652 ts_tablet_manager.cc:1398] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 14:13:47.950554 13449 raft_consensus.cc:482] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:47.950805 13449 raft_consensus.cc:504] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.951649 13449 leader_election.cc:288] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:47.952014 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616e17b094b74234b3c59bc13e655f95" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5" is_pre_election: true
I20240503 14:13:47.952240 9662 raft_consensus.cc:2454] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:47.952831 9733 leader_election.cc:302] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:47.953099 13127 raft_consensus.cc:2781] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:47.953202 13127 raft_consensus.cc:482] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:47.953300 13127 raft_consensus.cc:3037] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.953413 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616e17b094b74234b3c59bc13e655f95" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:47.953729 9528 raft_consensus.cc:2454] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:47.954290 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47877cdf5f04676861b3dc4708f3920" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:47.954627 9797 raft_consensus.cc:3037] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.956200 13127 raft_consensus.cc:504] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.957310 9662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616e17b094b74234b3c59bc13e655f95" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
I20240503 14:13:47.957556 9662 raft_consensus.cc:3037] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.959905 9662 raft_consensus.cc:2454] T 616e17b094b74234b3c59bc13e655f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:47.960923 9733 leader_election.cc:302] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:47.961923 13449 raft_consensus.cc:2781] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.962069 13449 raft_consensus.cc:686] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:47.962337 13449 consensus_queue.cc:237] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } }
I20240503 14:13:47.960755 13127 leader_election.cc:288] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:47.965898 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "616e17b094b74234b3c59bc13e655f95" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:47.966115 9528 raft_consensus.cc:3037] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.969087 9797 raft_consensus.cc:2454] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:47.970091 13282 leader_election.cc:288] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:47.970212 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f47877cdf5f04676861b3dc4708f3920" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:47.970458 7056 raft_consensus.cc:3037] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:47.970829 9864 leader_election.cc:302] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:47.971213 13282 raft_consensus.cc:2781] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:47.970997 6658 catalog_manager.cc:5484] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:47.971354 13282 raft_consensus.cc:686] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:47.971580 13282 consensus_queue.cc:237] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } }
I20240503 14:13:47.973479 6655 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 14:13:47.973591 7056 raft_consensus.cc:2454] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:47.974505 9528 raft_consensus.cc:2454] T 616e17b094b74234b3c59bc13e655f95 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:47.980679 7190 raft_consensus.cc:1261] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:47.981478 13380 consensus_queue.cc:1035] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.985158 9662 raft_consensus.cc:1261] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:47.986168 13644 consensus_queue.cc:1035] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:47.986145 6655 catalog_manager.cc:5484] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 0 to 1, leader changed from <none> to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:47.987731 13670 tablet.cc:1722] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:47.988591 6658 catalog_manager.cc:5576] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337) does not have the latest schema for tablet e14af3987fbc44e896615d5229d815d7 (table rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67]). Expected version 1 got 0
I20240503 14:13:47.988829 13670 tablet.cc:1620] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:47.998009 13671 tablet.cc:1722] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:47.999987 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67] alter complete (version 1)
I20240503 14:13:48.000065 13671 tablet.cc:1620] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:48.006306 6658 catalog_manager.cc:5576] TS e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337) does not have the latest schema for tablet e14af3987fbc44e896615d5229d815d7 (table rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67]). Expected version 1 got 0
I20240503 14:13:48.019307 13679 tablet.cc:1722] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:48.019971 13679 tablet.cc:1620] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:48.023650 6651 catalog_manager.cc:5576] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) does not have the latest schema for tablet e14af3987fbc44e896615d5229d815d7 (table rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67]). Expected version 1 got 0
I20240503 14:13:48.029651 12805 raft_consensus.cc:1050] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:48.030550 12805 consensus_queue.cc:237] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:48.042670 6798 raft_consensus.cc:1261] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:48.043730 13434 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:48.047113 9797 raft_consensus.cc:1261] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:48.047672 13434 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:48.047839 13449 raft_consensus.cc:482] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 14:13:48.048012 13449 raft_consensus.cc:504] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:48.048664 13449 leader_election.cc:288] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:48.049266 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d04b4d914bc485fa3f437ad0eb40cb5" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:48.049500 7190 raft_consensus.cc:2454] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:48.049649 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d04b4d914bc485fa3f437ad0eb40cb5" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:48.049850 7056 raft_consensus.cc:2454] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 0.
I20240503 14:13:48.050364 9735 leader_election.cc:302] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 3efdbd3506274dba846ae40404f0572c; no voters:
I20240503 14:13:48.050674 13449 raft_consensus.cc:2781] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 14:13:48.050778 13449 raft_consensus.cc:482] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 14:13:48.050886 13449 raft_consensus.cc:3037] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.054608 13449 raft_consensus.cc:504] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:48.055824 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d04b4d914bc485fa3f437ad0eb40cb5" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:48.056069 7056 raft_consensus.cc:3037] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.058738 9662 raft_consensus.cc:1261] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:48.059499 7056 raft_consensus.cc:2454] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:48.059448 12805 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:48.059962 9735 leader_election.cc:302] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 3efdbd3506274dba846ae40404f0572c; no voters:
I20240503 14:13:48.060317 13127 raft_consensus.cc:2781] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader election won for term 1
I20240503 14:13:48.060457 13127 raft_consensus.cc:686] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:48.060741 13127 consensus_queue.cc:237] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:48.061003 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d04b4d914bc485fa3f437ad0eb40cb5" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:48.061240 7190 raft_consensus.cc:3037] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708 [term 0 FOLLOWER]: Advancing to term 1
I20240503 14:13:48.068297 7190 raft_consensus.cc:2454] T 4d04b4d914bc485fa3f437ad0eb40cb5 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:48.069085 13449 leader_election.cc:288] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 1 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:48.074471 13434 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:48.076578 13568 raft_consensus.cc:1050] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:48.077427 13568 consensus_queue.cc:237] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:48.085896 6651 catalog_manager.cc:5484] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:48.098799 7056 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:48.099503 13572 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:48.102041 9797 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:48.102949 13572 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:48.105851 6651 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34184:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 14:13:48.117862 6658 catalog_manager.cc:5576] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609) does not have the latest schema for tablet e14af3987fbc44e896615d5229d815d7 (table rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67]). Expected version 2 got 1
I20240503 14:13:48.122623 13670 tablet.cc:1722] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:48.123152 13670 tablet.cc:1620] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: MemRowSet was empty: no flush needed.
I20240503 14:13:48.123430 13671 tablet.cc:1722] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:48.123940 13671 tablet.cc:1620] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708: MemRowSet was empty: no flush needed.
I20240503 14:13:48.132941 9929 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:48.134008 13572 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:48.142885 6651 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 371d064947984511bcd526e0de3fb910: rebalancer_dynamic_table [id=3e15496763ec4d60ab8a7db1e2d2fc67] alter complete (version 2)
I20240503 14:13:48.153873 13679 tablet.cc:1722] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:48.154392 13679 tablet.cc:1620] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5: MemRowSet was empty: no flush needed.
I20240503 14:13:48.167290 6798 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:48.168116 13549 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:48.202375 9797 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
W20240503 14:13:48.215226 6734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 43825.7us
I20240503 14:13:48.225571 6655 catalog_manager.cc:5484] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:48.227690 6798 consensus_queue.cc:237] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:48.230861 9930 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:48.231518 13573 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:48.234732 9930 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:48.242462 13572 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:48.249056 7056 consensus_queue.cc:237] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:48.257946 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3160f1a600604f258f76614f13429563 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:48.263283 6655 catalog_manager.cc:5484] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 2 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:48.267648 6799 raft_consensus.cc:1261] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:48.268733 6798 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:48.274024 9798 raft_consensus.cc:1261] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:48.277237 13703 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:48.277794 6928 raft_consensus.cc:1261] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:48.278015 9798 raft_consensus.cc:1261] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:48.278385 13568 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:48.278646 13573 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:48.280040 7056 raft_consensus.cc:1261] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:48.281078 13572 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:48.286034 9798 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:48.290597 13703 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:48.291296 13703 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:48.293779 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 3160f1a600604f258f76614f13429563 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:48.293933 7056 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:48.294593 13549 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:48.296037 6800 raft_consensus.cc:2932] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:48.296942 9795 raft_consensus.cc:1261] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:48.297521 6658 catalog_manager.cc:5484] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:48.298805 13573 consensus_queue.cc:1035] T 3160f1a600604f258f76614f13429563 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:48.300457 9795 raft_consensus.cc:2932] T 3160f1a600604f258f76614f13429563 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:48.303575 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet c59daa2d12f446899c7bb9970e1694da with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:48.320050 9664 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:48.320627 13549 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:48.330825 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c59daa2d12f446899c7bb9970e1694da with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:48.335728 9529 consensus_queue.cc:237] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:48.339357 6800 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:48.340296 9664 raft_consensus.cc:1261] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:48.341642 13127 consensus_queue.cc:1035] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.341863 13380 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:48.342376 13644 consensus_queue.cc:1035] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:48.347307 13380 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:48.350037 6800 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:48.356493 13568 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:48.358834 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6e0bb82edac2464b9d618c1c80807fb2 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:48.360486 6658 catalog_manager.cc:5484] T 6e0bb82edac2464b9d618c1c80807fb2 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:48.362874 9799 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:48.364542 9665 raft_consensus.cc:2932] T 6e0bb82edac2464b9d618c1c80807fb2 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:48.379302 13521 consensus_queue.cc:1035] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.383366 7040 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e0bb82edac2464b9d618c1c80807fb2 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:48.383796 13715 tablet_replica.cc:330] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:48.384068 13715 raft_consensus.cc:2229] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:48.384317 13715 pending_rounds.cc:62] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: Trying to abort 1 pending ops.
I20240503 14:13:48.384546 13715 pending_rounds.cc:68] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023598092410904576 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c59daa2d12f446899c7bb9970e1694da" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } } }
I20240503 14:13:48.385833 13715 raft_consensus.cc:2894] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 14:13:48.386016 13715 raft_consensus.cc:2866] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:48.386209 13715 raft_consensus.cc:2258] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:48.386787 6928 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:48.388448 13713 tablet_replica.cc:330] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:48.393007 6651 catalog_manager.cc:5484] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:48.399905 6800 consensus_queue.cc:237] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:48.404368 6928 raft_consensus.cc:1261] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:48.405237 13568 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:48.401006 7056 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:48.408869 13713 raft_consensus.cc:2229] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:48.409423 13713 pending_rounds.cc:62] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Trying to abort 1 pending ops.
I20240503 14:13:48.409265 13572 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:48.409752 13713 pending_rounds.cc:68] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Aborting op as it isn't in flight: id { term: 2 index: 9 } timestamp: 7023598090566832128 op_type: NO_OP noop_request { }
I20240503 14:13:48.409730 6928 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:48.413368 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8477be3143f64d83a4c68f260a66efbf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:48.410331 13713 raft_consensus.cc:2866] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:13:48.413498 7061 raft_consensus.cc:1261] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:48.414553 13713 raft_consensus.cc:2258] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:48.416129 13713 ts_tablet_manager.cc:1875] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:48.416662 6650 catalog_manager.cc:5484] T 8477be3143f64d83a4c68f260a66efbf P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:48.420179 13568 consensus_queue.cc:1035] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:48.422060 13715 ts_tablet_manager.cc:1875] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:48.423185 7061 raft_consensus.cc:2932] T 8477be3143f64d83a4c68f260a66efbf P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:48.427944 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 8477be3143f64d83a4c68f260a66efbf with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:48.440642 13723 tablet_replica.cc:330] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:48.441599 13723 raft_consensus.cc:2229] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:48.442179 13723 raft_consensus.cc:2258] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:48.444818 13282 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.453539 13723 ts_tablet_manager.cc:1875] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:48.459648 13713 ts_tablet_manager.cc:1888] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:13:48.459789 13713 log.cc:1196] T 3160f1a600604f258f76614f13429563 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/3160f1a600604f258f76614f13429563
I20240503 14:13:48.460498 13713 tablet_replica.cc:330] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:48.460551 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 3160f1a600604f258f76614f13429563 (table rebalance_test_table_19 [id=e5eb9ea1539d44d184cef03365a06046]) successfully deleted
I20240503 14:13:48.460687 13713 raft_consensus.cc:2229] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:48.460824 13713 pending_rounds.cc:62] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Trying to abort 1 pending ops.
I20240503 14:13:48.460863 13713 pending_rounds.cc:68] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Aborting op as it isn't in flight: id { term: 2 index: 9 } timestamp: 7023598091383103488 op_type: NO_OP noop_request { }
I20240503 14:13:48.460973 13713 raft_consensus.cc:2866] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 14:13:48.461052 13713 raft_consensus.cc:2258] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:48.462854 13715 ts_tablet_manager.cc:1888] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:48.463095 13715 log.cc:1196] T c59daa2d12f446899c7bb9970e1694da P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/c59daa2d12f446899c7bb9970e1694da
I20240503 14:13:48.464159 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet c59daa2d12f446899c7bb9970e1694da (table rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]) successfully deleted
I20240503 14:13:48.465317 13713 ts_tablet_manager.cc:1875] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:48.472258 13511 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.486219 13713 ts_tablet_manager.cc:1888] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:13:48.486338 13713 log.cc:1196] T 6e0bb82edac2464b9d618c1c80807fb2 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/6e0bb82edac2464b9d618c1c80807fb2
I20240503 14:13:48.487048 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 6e0bb82edac2464b9d618c1c80807fb2 (table rebalance_test_table_3 [id=d19eb7d5a50440f095be21549b58589b]) successfully deleted
I20240503 14:13:48.490139 13723 ts_tablet_manager.cc:1888] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:48.490273 13723 log.cc:1196] T 8477be3143f64d83a4c68f260a66efbf P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/8477be3143f64d83a4c68f260a66efbf
I20240503 14:13:48.491958 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 8477be3143f64d83a4c68f260a66efbf (table rebalancer_extra_table_14 [id=cc43ce3e383c426c9150416150a4895d]) successfully deleted
I20240503 14:13:48.508955 6928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb"
dest_uuid: "7186ed8510424fd597173eaf10c57b12"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41034
I20240503 14:13:48.509136 6928 raft_consensus.cc:595] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:48.560032 9929 consensus_queue.cc:237] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:48.562711 9529 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:48.563020 9799 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:48.563230 9863 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77d3f36de08d4f1986e42e86e1b95fbf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.563665 13511 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:48.564046 13509 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:48.567180 13511 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:48.567862 9799 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:48.569653 9529 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:48.577078 6658 catalog_manager.cc:5484] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:48.583230 13521 consensus_queue.cc:1035] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.613943 13449 consensus_queue.cc:1035] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 14:13:48.624500 13721 raft_consensus.cc:979] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: : Instructing follower 5b7e36bf354f480c90859cb8867b59cf to start an election
I20240503 14:13:48.624666 13721 raft_consensus.cc:1067] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Signalling peer 5b7e36bf354f480c90859cb8867b59cf to start an election
I20240503 14:13:48.625895 9529 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb"
dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
from {username='slave'} at 127.0.186.2:57189
I20240503 14:13:48.626114 9529 raft_consensus.cc:482] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:48.626259 9529 raft_consensus.cc:3037] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:48.629052 9529 raft_consensus.cc:504] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:48.630345 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:48.630564 6928 raft_consensus.cc:3032] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:48.630666 6928 raft_consensus.cc:729] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:48.630954 6928 consensus_queue.cc:260] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:48.631383 6928 raft_consensus.cc:3037] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:48.631570 9529 leader_election.cc:288] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Requested vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:48.634088 6928 raft_consensus.cc:2454] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:48.635658 9929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:48.635877 9929 raft_consensus.cc:3037] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:48.638119 9929 raft_consensus.cc:2454] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:48.640076 9467 leader_election.cc:302] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:48.640486 13380 raft_consensus.cc:2781] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:48.640642 13380 raft_consensus.cc:686] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:48.641045 13380 consensus_queue.cc:237] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:48.643461 9799 consensus_queue.cc:237] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:48.646162 7061 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:48.646813 13127 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:48.647869 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9027b8bfee0443aab7420df3f7257bcb" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:48.648149 7187 raft_consensus.cc:3037] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:48.648321 9929 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:48.648905 13127 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:48.649588 9733 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 01e27516acca49459176362caf73595e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.651168 7187 raft_consensus.cc:2454] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:48.652056 6658 catalog_manager.cc:5484] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 1 to 2, leader changed from 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:48.661445 13127 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:48.661810 7056 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:48.670522 6651 catalog_manager.cc:5484] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 4 to 5, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:48.673059 9929 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:48.716341 6928 consensus_queue.cc:237] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:48.728984 9929 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:48.729199 7056 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 14:13:48.729341 6867 consensus_peers.cc:485] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34f1ed0cfa8948048b3762919481367c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.729624 13237 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:48.729928 13721 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:48.734699 13237 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:48.736192 9929 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:48.740100 7061 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:48.748188 6658 catalog_manager.cc:5484] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:48.833009 9929 consensus_queue.cc:237] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:48.849141 9799 raft_consensus.cc:1261] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:48.849192 9665 raft_consensus.cc:1261] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:48.849579 9863 consensus_peers.cc:485] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c7281ccb4c44c8d8cebf78a1af6dd66. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.849915 13282 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:48.850234 13511 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:48.853345 13511 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:48.853832 9665 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:48.862872 6658 catalog_manager.cc:5484] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:48.882949 9799 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:48.897619 7187 consensus_queue.cc:237] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
W20240503 14:13:48.900166 7125 consensus_peers.cc:485] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 -> Peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): Couldn't send request to peer ea801601713a4d55aee0de9e3473e0bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4b4363edcc954769a3ab9c658e198c2d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:48.900157 6928 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:48.900290 9799 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:48.900851 13304 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:48.901146 13305 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:48.904646 13305 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:48.905153 6928 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:48.909714 9799 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:48.911685 6655 catalog_manager.cc:5484] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 6 to 7, NON_VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } } }
I20240503 14:13:48.943928 13804 ts_tablet_manager.cc:902] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:48.947417 13804 tablet_copy_client.cc:308] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:48.948179 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet 77d3f36de08d4f1986e42e86e1b95fbf from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:44495)
I20240503 14:13:48.948364 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet 77d3f36de08d4f1986e42e86e1b95fbf from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:44495: session id = f03dc37fe3234033bc665bed4f1b51f5-77d3f36de08d4f1986e42e86e1b95fbf
I20240503 14:13:48.950798 9950 tablet_copy_source_session.cc:205] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:48.952716 13804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77d3f36de08d4f1986e42e86e1b95fbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:48.959040 13804 tablet_copy_client.cc:783] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:48.959811 13804 tablet_copy_client.cc:647] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:48.961443 13804 tablet_copy_client.cc:523] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:48.965612 13804 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:48.972056 13804 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:48.974848 13804 tablet_bootstrap.cc:492] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:48.975180 13804 ts_tablet_manager.cc:1367] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 14:13:48.977582 13804 raft_consensus.cc:348] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:48.978698 13804 raft_consensus.cc:729] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:48.978904 13804 consensus_queue.cc:260] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:48.979535 13804 ts_tablet_manager.cc:1398] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 14:13:48.980649 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-77d3f36de08d4f1986e42e86e1b95fbf received from {username='slave'} at 127.0.186.6:44495
I20240503 14:13:48.980809 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-77d3f36de08d4f1986e42e86e1b95fbf on tablet 77d3f36de08d4f1986e42e86e1b95fbf with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:49.098503 9665 consensus_queue.cc:237] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:49.101068 7187 raft_consensus.cc:1261] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:49.101320 9799 raft_consensus.cc:1261] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:49.101773 9601 consensus_peers.cc:485] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62d262e208a84c14bd296e0c44947813. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:49.102133 13074 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:49.102558 13406 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:49.109249 13074 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:49.109820 9799 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:49.119408 6655 catalog_manager.cc:5484] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:49.120147 9929 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:49.120744 13380 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:49.128455 7187 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:49.138096 13882 ts_tablet_manager.cc:902] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:49.161509 7061 consensus_queue.cc:237] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:49.167925 9929 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:49.168494 13434 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:13:49.169680 6997 consensus_peers.cc:485] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8848a40563c43468ac46f8c604cc578. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:49.171831 7187 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:49.172260 13882 tablet_copy_client.cc:308] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:49.173049 13434 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:49.173799 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 01e27516acca49459176362caf73595e from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:50341)
I20240503 14:13:49.174048 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 01e27516acca49459176362caf73595e from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:50341: session id = 7186ed8510424fd597173eaf10c57b12-01e27516acca49459176362caf73595e
I20240503 14:13:49.177093 9819 tablet_copy_source_session.cc:205] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.180471 13703 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:49.181602 9929 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:49.188839 7190 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:49.189980 13644 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:49.189807 6655 catalog_manager.cc:5484] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:49.193750 13882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e27516acca49459176362caf73595e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.194769 7190 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:49.199231 6928 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:49.200055 13644 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:49.219072 13882 tablet_copy_client.cc:783] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.221597 13882 tablet_copy_client.cc:647] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.223274 13892 ts_tablet_manager.cc:902] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:49.227955 13882 tablet_copy_client.cc:523] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.243422 13892 tablet_copy_client.cc:308] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:49.244553 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 34f1ed0cfa8948048b3762919481367c from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:46745)
I20240503 14:13:49.244733 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 34f1ed0cfa8948048b3762919481367c from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:46745: session id = 5b7e36bf354f480c90859cb8867b59cf-34f1ed0cfa8948048b3762919481367c
I20240503 14:13:49.248346 6951 tablet_copy_source_session.cc:205] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.250001 13892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34f1ed0cfa8948048b3762919481367c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.265528 9529 consensus_queue.cc:237] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:49.268015 7187 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:49.268637 13380 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:49.275403 7190 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:49.287906 13892 tablet_copy_client.cc:783] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.291427 9529 consensus_queue.cc:237] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:49.292197 6655 catalog_manager.cc:5484] T 9027b8bfee0443aab7420df3f7257bcb P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 8 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:49.296976 13892 tablet_copy_client.cc:647] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.298508 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 9027b8bfee0443aab7420df3f7257bcb with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:49.305847 13882 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:49.328085 9930 raft_consensus.cc:1261] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:49.347409 13911 ts_tablet_manager.cc:902] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:49.348805 13913 tablet_replica.cc:330] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:49.350046 13913 raft_consensus.cc:2229] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:49.350383 13913 raft_consensus.cc:2258] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:49.353994 9467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 80615us
I20240503 14:13:49.355190 13882 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:49.356359 13882 tablet_bootstrap.cc:492] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:49.356645 13882 ts_tablet_manager.cc:1367] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.000s
I20240503 14:13:49.357412 13882 raft_consensus.cc:348] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:49.357668 13882 raft_consensus.cc:729] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:49.357689 9929 raft_consensus.cc:1261] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:49.357882 13882 consensus_queue.cc:260] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:49.359011 13380 consensus_queue.cc:1035] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:49.359843 13380 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:49.361120 13892 tablet_copy_client.cc:523] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.361343 13882 ts_tablet_manager.cc:1398] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 14:13:49.362428 9929 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:49.363447 13907 ts_tablet_manager.cc:902] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:49.364063 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-01e27516acca49459176362caf73595e received from {username='slave'} at 127.0.186.2:50341
I20240503 14:13:49.364192 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-01e27516acca49459176362caf73595e on tablet 01e27516acca49459176362caf73595e with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:49.364692 13892 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:49.367053 13913 ts_tablet_manager.cc:1875] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:49.380489 9799 raft_consensus.cc:1261] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:49.382119 13644 raft_consensus.cc:2932] T 9027b8bfee0443aab7420df3f7257bcb P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:49.382732 13903 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20240503 14:13:49.385267 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 10ad2f2ef32d4fc88cdd62bfeab97b8a. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:49.404875 13914 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:49.407235 9929 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:49.415675 6655 catalog_manager.cc:5484] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:49.418663 9799 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:49.424264 13913 ts_tablet_manager.cc:1888] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:13:49.426533 13913 log.cc:1196] T 9027b8bfee0443aab7420df3f7257bcb P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/9027b8bfee0443aab7420df3f7257bcb
I20240503 14:13:49.427497 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 9027b8bfee0443aab7420df3f7257bcb (table rebalance_test_table_13 [id=bec9593028ac497a88ec6126d31cde00]) successfully deleted
I20240503 14:13:49.428676 13907 tablet_copy_client.cc:308] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:49.429497 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:44207)
I20240503 14:13:49.429682 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:44207: session id = 7186ed8510424fd597173eaf10c57b12-8c7281ccb4c44c8d8cebf78a1af6dd66
I20240503 14:13:49.432531 9950 tablet_copy_source_session.cc:205] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.434198 13907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.444136 13911 tablet_copy_client.cc:308] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:49.445020 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 4b4363edcc954769a3ab9c658e198c2d from peer ea801601713a4d55aee0de9e3473e0bc ({username='slave'} at 127.0.186.8:41897)
I20240503 14:13:49.445183 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 4b4363edcc954769a3ab9c658e198c2d from peer ea801601713a4d55aee0de9e3473e0bc at {username='slave'} at 127.0.186.8:41897: session id = ea801601713a4d55aee0de9e3473e0bc-4b4363edcc954769a3ab9c658e198c2d
I20240503 14:13:49.446653 13907 tablet_copy_client.cc:783] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.447059 13907 tablet_copy_client.cc:647] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.448868 13907 tablet_copy_client.cc:523] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.448964 7211 tablet_copy_source_session.cc:205] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.452440 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9027b8bfee0443aab7420df3f7257bcb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:49.455049 13911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b4363edcc954769a3ab9c658e198c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.462404 13911 tablet_copy_client.cc:783] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.462777 13911 tablet_copy_client.cc:647] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.467900 13907 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:49.470039 9665 raft_consensus.cc:1203] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:49.509773 13892 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:49.510417 13892 tablet_bootstrap.cc:492] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:49.510823 13892 ts_tablet_manager.cc:1367] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.146s user 0.057s sys 0.004s
I20240503 14:13:49.511986 13892 raft_consensus.cc:348] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:49.512418 13892 raft_consensus.cc:729] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:49.512696 13892 consensus_queue.cc:260] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:49.513764 13892 ts_tablet_manager.cc:1398] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:49.515271 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-34f1ed0cfa8948048b3762919481367c received from {username='slave'} at 127.0.186.5:46745
I20240503 14:13:49.515412 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-34f1ed0cfa8948048b3762919481367c on tablet 34f1ed0cfa8948048b3762919481367c with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:49.521562 13911 tablet_copy_client.cc:523] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.535091 13911 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: Bootstrap starting.
I20240503 14:13:49.535682 13907 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:49.536073 13907 tablet_bootstrap.cc:492] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:49.536334 13907 ts_tablet_manager.cc:1367] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.069s user 0.020s sys 0.030s
I20240503 14:13:49.537128 13907 raft_consensus.cc:348] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:49.537413 13907 raft_consensus.cc:729] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:49.537613 13907 consensus_queue.cc:260] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:49.538219 13907 ts_tablet_manager.cc:1398] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:49.539541 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-8c7281ccb4c44c8d8cebf78a1af6dd66 received from {username='slave'} at 127.0.186.2:44207
I20240503 14:13:49.539733 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-8c7281ccb4c44c8d8cebf78a1af6dd66 on tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:49.547194 6931 raft_consensus.cc:1203] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:49.596765 13911 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:49.597169 13911 tablet_bootstrap.cc:492] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: Bootstrap complete.
I20240503 14:13:49.597465 13911 ts_tablet_manager.cc:1367] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: Time spent bootstrapping tablet: real 0.062s user 0.048s sys 0.001s
I20240503 14:13:49.598166 13911 raft_consensus.cc:348] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:49.598387 13911 raft_consensus.cc:729] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Initialized, Role: LEARNER
I20240503 14:13:49.598634 13911 consensus_queue.cc:260] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: NON_VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: true } }
I20240503 14:13:49.599174 13911 ts_tablet_manager.cc:1398] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:49.599956 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session ea801601713a4d55aee0de9e3473e0bc-4b4363edcc954769a3ab9c658e198c2d received from {username='slave'} at 127.0.186.8:41897
I20240503 14:13:49.600075 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session ea801601713a4d55aee0de9e3473e0bc-4b4363edcc954769a3ab9c658e198c2d on tablet 4b4363edcc954769a3ab9c658e198c2d with peer ea801601713a4d55aee0de9e3473e0bc
I20240503 14:13:49.613632 13892 ts_tablet_manager.cc:902] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:49.631361 13967 ts_tablet_manager.cc:902] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:49.631919 13892 tablet_copy_client.cc:308] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:49.632679 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet c8848a40563c43468ac46f8c604cc578 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:53967)
I20240503 14:13:49.632838 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet c8848a40563c43468ac46f8c604cc578 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:53967: session id = 5b7e36bf354f480c90859cb8867b59cf-c8848a40563c43468ac46f8c604cc578
I20240503 14:13:49.636152 7081 tablet_copy_source_session.cc:205] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.652761 13892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8848a40563c43468ac46f8c604cc578. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.675931 13892 tablet_copy_client.cc:783] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.676530 13892 tablet_copy_client.cc:647] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.677274 13967 tablet_copy_client.cc:308] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.6:33971
I20240503 14:13:49.678104 9685 tablet_copy_service.cc:140] P f03dc37fe3234033bc665bed4f1b51f5: Received BeginTabletCopySession request for tablet 62d262e208a84c14bd296e0c44947813 from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:35985)
I20240503 14:13:49.678315 9685 tablet_copy_service.cc:161] P f03dc37fe3234033bc665bed4f1b51f5: Beginning new tablet copy session on tablet 62d262e208a84c14bd296e0c44947813 from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:35985: session id = 09723f236d8044538d920a37b5d9aa8c-62d262e208a84c14bd296e0c44947813
I20240503 14:13:49.678553 13892 tablet_copy_client.cc:523] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.681430 9685 tablet_copy_source_session.cc:205] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.682413 13892 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
I20240503 14:13:49.685956 13967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d262e208a84c14bd296e0c44947813. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.698554 13967 tablet_copy_client.cc:783] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.698949 13967 tablet_copy_client.cc:647] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.701166 13967 tablet_copy_client.cc:523] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.705004 13967 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:49.716248 13967 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:49.717053 13967 tablet_bootstrap.cc:492] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:49.717897 13967 ts_tablet_manager.cc:1367] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20240503 14:13:49.718994 13967 raft_consensus.cc:348] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:49.720147 13967 raft_consensus.cc:729] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:49.720523 13967 consensus_queue.cc:260] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:49.721544 13967 ts_tablet_manager.cc:1398] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:49.722931 9685 tablet_copy_service.cc:342] P f03dc37fe3234033bc665bed4f1b51f5: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-62d262e208a84c14bd296e0c44947813 received from {username='slave'} at 127.0.186.3:35985
I20240503 14:13:49.723114 9685 tablet_copy_service.cc:433] P f03dc37fe3234033bc665bed4f1b51f5: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-62d262e208a84c14bd296e0c44947813 on tablet 62d262e208a84c14bd296e0c44947813 with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:49.797960 6931 raft_consensus.cc:1203] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:49.804201 9532 raft_consensus.cc:1203] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:49.807117 13892 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:49.807765 13892 tablet_bootstrap.cc:492] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:49.808125 13892 ts_tablet_manager.cc:1367] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.126s user 0.052s sys 0.033s
I20240503 14:13:49.809090 13892 raft_consensus.cc:348] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:49.809448 13892 raft_consensus.cc:729] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:49.809744 13892 consensus_queue.cc:260] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:49.810729 13892 ts_tablet_manager.cc:1398] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 14:13:49.811612 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-c8848a40563c43468ac46f8c604cc578 received from {username='slave'} at 127.0.186.5:53967
I20240503 14:13:49.811748 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-c8848a40563c43468ac46f8c604cc578 on tablet c8848a40563c43468ac46f8c604cc578 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:49.827970 9929 raft_consensus.cc:1203] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:49.952488 13967 ts_tablet_manager.cc:902] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:49.959700 13967 tablet_copy_client.cc:308] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.5:36609
I20240503 14:13:49.960589 9553 tablet_copy_service.cc:140] P 5b7e36bf354f480c90859cb8867b59cf: Received BeginTabletCopySession request for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:55131)
I20240503 14:13:49.960822 9553 tablet_copy_service.cc:161] P 5b7e36bf354f480c90859cb8867b59cf: Beginning new tablet copy session on tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:55131: session id = 09723f236d8044538d920a37b5d9aa8c-10ad2f2ef32d4fc88cdd62bfeab97b8a
I20240503 14:13:49.963369 9553 tablet_copy_source_session.cc:205] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:49.964705 13967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:49.969422 13967 tablet_copy_client.cc:783] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:49.969676 13967 tablet_copy_client.cc:647] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:49.971390 13967 tablet_copy_client.cc:523] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:49.975139 13967 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:49.980650 13967 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:49.981195 13967 tablet_bootstrap.cc:492] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:49.981477 13967 ts_tablet_manager.cc:1367] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.006s user 0.007s sys 0.000s
I20240503 14:13:49.982100 13967 raft_consensus.cc:348] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:49.982317 13967 raft_consensus.cc:729] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:49.982507 13967 consensus_queue.cc:260] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:49.983150 13967 ts_tablet_manager.cc:1398] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 14:13:49.984119 9553 tablet_copy_service.cc:342] P 5b7e36bf354f480c90859cb8867b59cf: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-10ad2f2ef32d4fc88cdd62bfeab97b8a received from {username='slave'} at 127.0.186.3:55131
I20240503 14:13:49.984268 9553 tablet_copy_service.cc:433] P 5b7e36bf354f480c90859cb8867b59cf: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-10ad2f2ef32d4fc88cdd62bfeab97b8a on tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:50.016870 13282 raft_consensus.cc:1050] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:50.017724 13282 consensus_queue.cc:237] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:50.021010 9662 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.021082 9532 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.021716 13509 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.022177 9799 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.022200 13511 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.023046 13511 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.026748 13511 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:50.027171 9662 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:50.027366 9799 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:50.031415 9532 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:50.034016 6655 catalog_manager.cc:5484] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:50.151360 13127 raft_consensus.cc:1050] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:50.152484 13127 consensus_queue.cc:237] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:50.156961 6931 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:50.157142 7056 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:50.157748 14045 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:50.158046 9928 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:50.158125 14043 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:50.158730 14043 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:50.166055 13768 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.168040 7056 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.173769 6651 catalog_manager.cc:5484] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:50.175176 6931 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.174144 9928 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.187261 9529 raft_consensus.cc:1203] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 14:13:50.191134 13509 raft_consensus.cc:1050] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:50.192005 13509 consensus_queue.cc:237] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:50.201907 6931 raft_consensus.cc:1261] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.202986 9797 raft_consensus.cc:1261] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.206332 9665 raft_consensus.cc:1261] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.207072 13282 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.207665 13282 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.209225 13282 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.231375 7056 raft_consensus.cc:1203] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:50.232228 13509 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.240034 9665 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.242592 6655 catalog_manager.cc:5484] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:50.246176 9799 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.251534 6931 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.258395 9928 consensus_queue.cc:237] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:50.261214 9799 raft_consensus.cc:1261] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:50.261780 6928 raft_consensus.cc:1261] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:50.261752 13282 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:50.262647 13282 consensus_queue.cc:1035] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:50.265426 13282 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.268016 9797 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.272071 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:50.272311 6928 raft_consensus.cc:2932] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.275729 6659 catalog_manager.cc:5484] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:50.279889 9644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 with delete_type TABLET_DATA_TOMBSTONED (TS f03dc37fe3234033bc665bed4f1b51f5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47036
I20240503 14:13:50.292151 14059 tablet_replica.cc:330] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: stopping tablet replica
I20240503 14:13:50.293534 14059 raft_consensus.cc:2229] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:50.293859 14059 raft_consensus.cc:2258] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:50.295020 14059 ts_tablet_manager.cc:1875] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:50.321406 13304 raft_consensus.cc:1050] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER ea801601713a4d55aee0de9e3473e0bc to VOTER
I20240503 14:13:50.322664 13304 consensus_queue.cc:237] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:50.333118 14059 ts_tablet_manager.cc:1888] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:50.333266 14059 log.cc:1196] T 8c7281ccb4c44c8d8cebf78a1af6dd66 P f03dc37fe3234033bc665bed4f1b51f5: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-5/wal/wals/8c7281ccb4c44c8d8cebf78a1af6dd66
I20240503 14:13:50.334141 6644 catalog_manager.cc:4830] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): tablet 8c7281ccb4c44c8d8cebf78a1af6dd66 (table rebalancer_extra_table_18 [id=6d7f4e3d99dd445fa6eccca10918b54f]) successfully deleted
I20240503 14:13:50.340343 9797 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.341041 9929 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.341465 13797 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:50.343060 6928 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.345619 13797 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:50.346612 13798 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:50.380731 13797 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:50.384235 13721 raft_consensus.cc:1050] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:50.386780 6931 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:50.386111 13721 consensus_queue.cc:237] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:50.420507 6659 catalog_manager.cc:5484] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:50.422863 9797 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
W20240503 14:13:50.435835 6735 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.7:42783, user_credentials={real_user=slave}} blocked reactor thread for 89909.1us
I20240503 14:13:50.436655 9929 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:50.455235 7190 consensus_queue.cc:237] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:50.459689 9799 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:50.460552 13798 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:50.465878 9929 raft_consensus.cc:1261] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:50.465996 7056 raft_consensus.cc:1203] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:50.470494 13797 consensus_queue.cc:1035] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:50.474298 13304 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:50.477348 9799 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:50.478333 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4b4363edcc954769a3ab9c658e198c2d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:50.479326 9928 raft_consensus.cc:2932] T 4b4363edcc954769a3ab9c658e198c2d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:50.491228 6659 catalog_manager.cc:5484] T 4b4363edcc954769a3ab9c658e198c2d P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 8 to 9, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } }
I20240503 14:13:50.506417 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b4363edcc954769a3ab9c658e198c2d with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:50.510454 9529 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.511669 9928 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.512915 14071 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20240503 14:13:50.513474 14073 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:50.535753 7056 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.537765 13721 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:50.541616 14087 tablet_replica.cc:330] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:50.541992 14087 raft_consensus.cc:2229] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:50.542311 14087 raft_consensus.cc:2258] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:50.547143 14087 ts_tablet_manager.cc:1875] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:50.556422 14088 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.557101 7056 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.558387 14087 ts_tablet_manager.cc:1888] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:13:50.558599 14087 log.cc:1196] T 4b4363edcc954769a3ab9c658e198c2d P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b4363edcc954769a3ab9c658e198c2d
I20240503 14:13:50.559741 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 4b4363edcc954769a3ab9c658e198c2d (table rebalance_test_table_15 [id=1e71137ec0ec4b3e9918c5ec1dc45b0d]) successfully deleted
I20240503 14:13:50.563508 9928 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.564230 6658 catalog_manager.cc:5484] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.564165 9529 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.577586 6931 consensus_queue.cc:237] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:50.579921 9529 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:50.579962 9928 raft_consensus.cc:1261] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 14:13:50.580462 14088 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:50.580869 14071 consensus_queue.cc:1035] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:50.585053 14088 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.585772 9529 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.590914 9929 raft_consensus.cc:2932] T 34f1ed0cfa8948048b3762919481367c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.591956 6651 catalog_manager.cc:5484] T 34f1ed0cfa8948048b3762919481367c P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 8 to 9, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.593014 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 34f1ed0cfa8948048b3762919481367c with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:50.596865 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 34f1ed0cfa8948048b3762919481367c with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:50.608989 14100 tablet_replica.cc:330] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:50.609330 14100 raft_consensus.cc:2229] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:50.609602 14100 raft_consensus.cc:2258] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:50.610656 14100 ts_tablet_manager.cc:1875] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:50.617687 14100 ts_tablet_manager.cc:1888] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 14:13:50.618016 14100 log.cc:1196] T 34f1ed0cfa8948048b3762919481367c P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/34f1ed0cfa8948048b3762919481367c
I20240503 14:13:50.619249 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 34f1ed0cfa8948048b3762919481367c (table rebalance_test_table_11 [id=c560a494affa4d7ab56273c13e488583]) successfully deleted
I20240503 14:13:50.651263 14057 raft_consensus.cc:1050] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:50.652367 14057 consensus_queue.cc:237] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:50.671263 7056 raft_consensus.cc:1261] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.673552 14010 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.675340 7190 raft_consensus.cc:1261] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.675868 9799 raft_consensus.cc:1261] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:50.676301 13874 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.677417 13874 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:50.711308 13874 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:50.722714 7056 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:50.725558 7190 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:50.734308 6651 catalog_manager.cc:5484] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:50.754504 9662 consensus_queue.cc:237] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:50.757743 7056 raft_consensus.cc:1261] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:50.758055 7190 raft_consensus.cc:1261] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:50.758878 14010 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:50.765448 14057 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:50.766386 7056 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:50.768980 14106 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:50.772569 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 62d262e208a84c14bd296e0c44947813 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:50.774009 7187 raft_consensus.cc:2932] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
W20240503 14:13:50.780725 6644 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 62d262e208a84c14bd296e0c44947813 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20240503 14:13:50.782660 6655 catalog_manager.cc:5484] T 62d262e208a84c14bd296e0c44947813 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:50.787881 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d262e208a84c14bd296e0c44947813 with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:50.826833 14117 tablet_replica.cc:330] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:50.826902 14114 raft_consensus.cc:1050] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:50.827198 14117 raft_consensus.cc:2229] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:50.827493 14117 pending_rounds.cc:62] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: Trying to abort 1 pending ops.
I20240503 14:13:50.827683 14117 pending_rounds.cc:68] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023598103149137920 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "62d262e208a84c14bd296e0c44947813" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } } }
I20240503 14:13:50.828253 14117 raft_consensus.cc:2894] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 14:13:50.828425 14117 raft_consensus.cc:2866] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:50.828209 14114 consensus_queue.cc:237] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:50.828610 14117 raft_consensus.cc:2258] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:50.828986 14109 consensus_queue.cc:1159] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "3efdbd3506274dba846ae40404f0572c" status { last_received { term: 1 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 3 } }
I20240503 14:13:50.835209 14117 ts_tablet_manager.cc:1875] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:50.843129 14117 ts_tablet_manager.cc:1888] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:50.843398 14117 log.cc:1196] T 62d262e208a84c14bd296e0c44947813 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/62d262e208a84c14bd296e0c44947813
I20240503 14:13:50.848033 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 62d262e208a84c14bd296e0c44947813 (table rebalancer_extra_table_10 [id=62c834b680c541d28239ce78af13d0f2]) successfully deleted
I20240503 14:13:50.861296 14083 raft_consensus.cc:1050] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:50.862640 14083 consensus_queue.cc:237] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:50.877380 9929 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.877429 9528 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.878458 14120 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:50.878958 13434 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:50.912608 7187 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 14:13:50.916560 14113 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:50.932701 13434 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.947721 9929 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.963567 6658 catalog_manager.cc:5484] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:50.968339 7187 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:51.005542 9528 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:51.027052 7061 raft_consensus.cc:1261] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:51.027115 9799 raft_consensus.cc:1261] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:51.027705 14074 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:51.028832 9929 raft_consensus.cc:1261] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:51.029755 13380 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:51.033512 13380 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:51.039826 14074 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.044317 9797 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.059684 9928 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.060037 7061 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.068238 6655 catalog_manager.cc:5484] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.069015 9929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36156
I20240503 14:13:51.069222 9929 raft_consensus.cc:595] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:51.076131 9528 consensus_queue.cc:237] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:51.078400 7056 raft_consensus.cc:1261] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:51.079182 14083 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:51.083915 14130 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.084789 7056 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.089567 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:51.092164 6659 catalog_manager.cc:5484] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.092561 9799 raft_consensus.cc:1261] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:51.093398 13380 consensus_queue.cc:1035] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:51.098567 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:51.095297 9797 raft_consensus.cc:2932] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:51.101145 14140 tablet_replica.cc:330] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:51.101562 14140 raft_consensus.cc:2229] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:51.103842 14140 raft_consensus.cc:2258] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:51.104753 14140 ts_tablet_manager.cc:1875] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:51.113075 14140 ts_tablet_manager.cc:1888] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:51.113272 14140 log.cc:1196] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/10ad2f2ef32d4fc88cdd62bfeab97b8a
I20240503 14:13:51.114193 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 10ad2f2ef32d4fc88cdd62bfeab97b8a (table rebalancer_extra_table_28 [id=a3664bdc6a0b47c5a4c912eeca5eb74b]) successfully deleted
I20240503 14:13:51.120509 9797 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01e27516acca49459176362caf73595e"
dest_uuid: "3efdbd3506274dba846ae40404f0572c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49192
I20240503 14:13:51.120819 9797 raft_consensus.cc:595] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:51.167397 13127 raft_consensus.cc:979] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: : Instructing follower ea801601713a4d55aee0de9e3473e0bc to start an election
I20240503 14:13:51.167645 13127 raft_consensus.cc:1067] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Signalling peer ea801601713a4d55aee0de9e3473e0bc to start an election
I20240503 14:13:51.168605 9928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01e27516acca49459176362caf73595e"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
from {username='slave'} at 127.0.186.7:59955
I20240503 14:13:51.168862 9928 raft_consensus.cc:482] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:51.169000 9928 raft_consensus.cc:3037] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.171401 9928 raft_consensus.cc:504] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:51.172303 9928 leader_election.cc:288] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:51.172587 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:51.172618 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:51.172626 6928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e27516acca49459176362caf73595e" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:51.172784 6928 raft_consensus.cc:3037] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.172791 7056 raft_consensus.cc:3037] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.172828 9797 raft_consensus.cc:3032] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:51.172916 9797 raft_consensus.cc:729] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:51.173262 9797 consensus_queue.cc:260] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:51.173693 9797 raft_consensus.cc:3037] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.174608 6928 raft_consensus.cc:2454] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:51.175623 7056 raft_consensus.cc:2454] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:51.176388 9797 raft_consensus.cc:2454] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:51.176976 9864 leader_election.cc:302] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:51.177313 13509 raft_consensus.cc:2781] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:51.177471 13509 raft_consensus.cc:686] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:51.177685 13509 consensus_queue.cc:237] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:51.183009 6658 catalog_manager.cc:5484] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 1 to 2, leader changed from 3efdbd3506274dba846ae40404f0572c (127.0.186.7) to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:51.196254 7056 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578"
dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39936
I20240503 14:13:51.196493 7056 raft_consensus.cc:595] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:51.264375 9928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36192
I20240503 14:13:51.264645 9928 raft_consensus.cc:595] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:51.306957 7056 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578"
dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39960
I20240503 14:13:51.307197 7056 raft_consensus.cc:595] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:51.334743 9928 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36214
I20240503 14:13:51.334964 9928 raft_consensus.cc:595] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:51.346834 13434 raft_consensus.cc:979] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: : Instructing follower ea801601713a4d55aee0de9e3473e0bc to start an election
I20240503 14:13:51.346998 13434 raft_consensus.cc:1067] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Signalling peer ea801601713a4d55aee0de9e3473e0bc to start an election
I20240503 14:13:51.347692 9928 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578"
dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
from {username='slave'} at 127.0.186.3:32947
I20240503 14:13:51.347878 9928 raft_consensus.cc:482] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:51.348006 9928 raft_consensus.cc:3037] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.350123 9928 raft_consensus.cc:504] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:51.351069 9928 leader_election.cc:288] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Requested vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:51.351560 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:51.351802 9528 raft_consensus.cc:3037] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.351814 7056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:51.352041 7056 raft_consensus.cc:3032] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:51.352136 7056 raft_consensus.cc:729] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:51.352257 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8848a40563c43468ac46f8c604cc578" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:51.352460 7187 raft_consensus.cc:3037] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.352398 7056 consensus_queue.cc:260] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:51.352913 7056 raft_consensus.cc:3037] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.354736 9528 raft_consensus.cc:2454] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:51.354903 7056 raft_consensus.cc:2454] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:51.355371 9865 leader_election.cc:302] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:51.355749 13509 raft_consensus.cc:2781] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:51.355888 13509 raft_consensus.cc:686] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:51.356184 13509 consensus_queue.cc:237] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:51.356791 7187 raft_consensus.cc:2454] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:51.360410 6658 catalog_manager.cc:5484] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:51.455850 13282 raft_consensus.cc:979] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: : Instructing follower f03dc37fe3234033bc665bed4f1b51f5 to start an election
I20240503 14:13:51.456573 13282 raft_consensus.cc:1067] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Signalling peer f03dc37fe3234033bc665bed4f1b51f5 to start an election
I20240503 14:13:51.457245 9665 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf"
dest_uuid: "f03dc37fe3234033bc665bed4f1b51f5"
from {username='slave'} at 127.0.186.8:35045
I20240503 14:13:51.457417 9665 raft_consensus.cc:482] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:51.457522 9665 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.459553 9665 raft_consensus.cc:504] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:51.460501 9665 leader_election.cc:288] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 2 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:51.460856 9797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:51.461086 9797 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.461356 9529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:51.461552 9529 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.464016 9797 raft_consensus.cc:2454] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 2.
I20240503 14:13:51.464278 9799 raft_consensus.cc:1226] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Rejecting Update request from peer ea801601713a4d55aee0de9e3473e0bc for earlier term 1. Current term is 2. Ops: []
I20240503 14:13:51.464372 9529 raft_consensus.cc:2454] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 2.
I20240503 14:13:51.464957 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "f03dc37fe3234033bc665bed4f1b51f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:51.465021 13282 consensus_queue.cc:1046] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:51.465431 9928 raft_consensus.cc:3032] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:51.465615 9928 raft_consensus.cc:729] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:51.466184 9928 consensus_queue.cc:260] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:51.466447 9600 leader_election.cc:302] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5; no voters:
I20240503 14:13:51.466751 9928 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:51.466738 14057 raft_consensus.cc:2781] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:51.466933 14057 raft_consensus.cc:686] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEADER]: Becoming Leader. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Running, Role: LEADER
I20240503 14:13:51.467200 14057 consensus_queue.cc:237] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:51.481472 9928 raft_consensus.cc:2454] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f03dc37fe3234033bc665bed4f1b51f5 in term 2.
W20240503 14:13:51.481719 13509 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20240503 14:13:51.484375 6658 catalog_manager.cc:5484] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: term changed from 1 to 2, leader changed from ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) to f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6). New cstate: current_term: 2 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:51.605229 9797 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:51.606047 13282 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:51.772294 6928 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:51.772887 13282 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:51.790374 7056 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:51.791100 13282 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:51.861104 9529 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:51.862250 14238 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:51.882323 7056 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:51.883155 14238 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:51.890303 7187 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 14:13:51.898896 13282 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 14:13:51.903316 9797 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:51.904912 14057 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:51.922269 9928 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:51.923485 14057 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:51.974411 9529 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:51.975122 14215 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:52.246908 9929 consensus_queue.cc:237] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:52.248878 6931 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:13:52.249099 7056 raft_consensus.cc:1261] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:13:52.250548 13509 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:52.252286 14238 consensus_queue.cc:1035] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:13:52.256564 14238 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:52.257545 7056 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:52.261786 6928 raft_consensus.cc:2932] T 01e27516acca49459176362caf73595e P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:52.264415 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 01e27516acca49459176362caf73595e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:52.266439 6658 catalog_manager.cc:5484] T 01e27516acca49459176362caf73595e P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 6 to 8, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:52.288460 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e27516acca49459176362caf73595e with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:52.292165 14285 tablet_replica.cc:330] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:52.292629 14285 raft_consensus.cc:2229] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:52.292984 14285 raft_consensus.cc:2258] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.294438 14285 ts_tablet_manager.cc:1875] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:52.308532 14285 ts_tablet_manager.cc:1888] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 14:13:52.308816 14285 log.cc:1196] T 01e27516acca49459176362caf73595e P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/01e27516acca49459176362caf73595e
I20240503 14:13:52.309991 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 01e27516acca49459176362caf73595e (table rebalancer_extra_table_0 [id=dd2af8958e8d41229b3619b42ff7c3a1]) successfully deleted
I20240503 14:13:52.370064 9928 consensus_queue.cc:237] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:52.371701 7190 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:52.371775 9529 raft_consensus.cc:1261] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 14:13:52.372296 14238 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:52.372604 13282 consensus_queue.cc:1035] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 14:13:52.374792 13282 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:52.375815 7190 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:52.376142 9529 raft_consensus.cc:2932] T c8848a40563c43468ac46f8c604cc578 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:52.379702 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8848a40563c43468ac46f8c604cc578 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:52.382958 6655 catalog_manager.cc:5484] T c8848a40563c43468ac46f8c604cc578 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 8 to 10, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:52.386528 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet c8848a40563c43468ac46f8c604cc578 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:52.388177 14296 tablet_replica.cc:330] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:52.388437 14296 raft_consensus.cc:2229] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:52.389034 14296 raft_consensus.cc:2258] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.390280 14296 ts_tablet_manager.cc:1875] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:52.395717 14296 ts_tablet_manager.cc:1888] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 14:13:52.395828 14296 log.cc:1196] T c8848a40563c43468ac46f8c604cc578 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/c8848a40563c43468ac46f8c604cc578
I20240503 14:13:52.396513 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet c8848a40563c43468ac46f8c604cc578 (table rebalance_test_table_5 [id=2266e15d2d8e4d4d9bc0b60ddc9a9e82]) successfully deleted
I20240503 14:13:52.502398 9664 consensus_queue.cc:237] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:52.504622 9529 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:52.504858 9797 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:52.505204 14057 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:52.505563 14215 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:52.508121 14215 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:52.508882 9797 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:52.513060 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 77d3f36de08d4f1986e42e86e1b95fbf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:52.512105 9529 raft_consensus.cc:2932] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:52.517081 6659 catalog_manager.cc:5484] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 5, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 2 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:52.521055 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 77d3f36de08d4f1986e42e86e1b95fbf with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:52.521925 14298 tablet_replica.cc:330] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:52.522135 14298 raft_consensus.cc:2229] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:52.522349 14298 raft_consensus.cc:2258] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:52.523073 14298 ts_tablet_manager.cc:1875] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:52.528607 14298 ts_tablet_manager.cc:1888] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 14:13:52.528808 14298 log.cc:1196] T 77d3f36de08d4f1986e42e86e1b95fbf P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/77d3f36de08d4f1986e42e86e1b95fbf
I20240503 14:13:52.529718 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 77d3f36de08d4f1986e42e86e1b95fbf (table rebalancer_extra_table_20 [id=2ed2516cb08e47538b24cba0a800484c]) successfully deleted
I20240503 14:13:52.988662 7039 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 14:13:53.020476 9779 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 14:13:53.020980 6911 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 14:13:53.028632 9513 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:13:53.075649 7170 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 14:13:53.122294 9909 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240503 14:13:53.179296 9644 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:13:53.199106 6780 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 14:13:53.367372 6931 consensus_queue.cc:237] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:53.369827 9928 raft_consensus.cc:1261] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.370101 7056 raft_consensus.cc:1261] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.370404 14379 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.370790 14071 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:53.371361 6867 consensus_peers.cc:485] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9a0d6f38293413aba812c473ab0973e. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.373399 14379 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:53.374020 9928 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:53.374874 7056 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:53.381494 6654 catalog_manager.cc:5484] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:53.410506 7056 consensus_queue.cc:237] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
W20240503 14:13:53.413316 6996 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5895d998bfb3439386fcbc2c74f8d9a6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.413305 6931 raft_consensus.cc:1261] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.413908 14377 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.414368 9529 raft_consensus.cc:1261] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.415023 14377 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.416973 14114 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:53.417588 6931 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:53.424335 9529 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:53.428611 6651 catalog_manager.cc:5484] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index -1 to 2, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:53.446940 9662 consensus_queue.cc:237] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:53.449807 9928 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:53.450030 9599 consensus_peers.cc:485] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3f2bd341493482f9519e6dd74cd2f3d. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.450294 9529 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.450532 14057 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.450800 14215 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.454963 9929 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.455212 9529 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.455698 14410 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.462024 6654 catalog_manager.cc:5484] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.480043 9799 consensus_queue.cc:237] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:53.482604 9529 raft_consensus.cc:1261] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:53.483248 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5dc7cefd974149148dcf856a50ea168c. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.483388 9929 raft_consensus.cc:1261] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.483431 14045 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.483955 13127 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.486476 14045 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:53.487095 9529 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:53.488337 9929 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:53.493294 6651 catalog_manager.cc:5484] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:53.525404 6931 consensus_queue.cc:237] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:53.527850 9929 raft_consensus.cc:1261] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.528443 14389 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.528759 9532 raft_consensus.cc:1261] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.529721 14389 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:53.530274 6866 consensus_peers.cc:485] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61b62d3aa0bc46c3be67eadc21466ec6. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.533047 14389 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:53.533836 9928 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:53.536346 9532 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:53.540177 6652 catalog_manager.cc:5484] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:53.563280 6931 consensus_queue.cc:237] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
W20240503 14:13:53.566291 6865 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 933c08460b9745feab9bcb56c27bdbb9. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.566458 6800 raft_consensus.cc:1261] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.566640 7056 raft_consensus.cc:1261] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.567019 14389 consensus_queue.cc:1035] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.567301 14380 consensus_queue.cc:1035] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.570704 14379 raft_consensus.cc:2932] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:53.570751 6800 raft_consensus.cc:2932] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:53.571952 7059 raft_consensus.cc:2932] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:53.577085 6651 catalog_manager.cc:5484] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:53.606451 9928 consensus_queue.cc:237] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:53.613526 9797 raft_consensus.cc:1261] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.613873 7059 raft_consensus.cc:1261] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.615067 14238 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:53.616353 9863 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f47877cdf5f04676861b3dc4708f3920. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.618347 14238 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.621697 14354 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.622344 7059 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.628937 6652 catalog_manager.cc:5484] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.650525 9799 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.710047 6800 consensus_queue.cc:237] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:53.713290 9928 raft_consensus.cc:1261] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.713882 9532 raft_consensus.cc:1261] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.714495 14336 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:53.714782 6734 consensus_peers.cc:485] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1df9d5b8a3d4a489cbccca67b223168. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.722205 14336 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.726263 14336 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:53.730381 9532 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:53.740839 9928 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:53.744161 6652 catalog_manager.cc:5484] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:53.810457 14461 ts_tablet_manager.cc:902] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:53.828616 14461 tablet_copy_client.cc:308] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:53.829432 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet b9a0d6f38293413aba812c473ab0973e from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:46745)
I20240503 14:13:53.829599 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet b9a0d6f38293413aba812c473ab0973e from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:46745: session id = 5b7e36bf354f480c90859cb8867b59cf-b9a0d6f38293413aba812c473ab0973e
I20240503 14:13:53.833030 6951 tablet_copy_source_session.cc:205] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.834506 14461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9a0d6f38293413aba812c473ab0973e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.840596 14461 tablet_copy_client.cc:783] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.841459 14461 tablet_copy_client.cc:647] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.842237 9532 consensus_queue.cc:237] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:53.843209 14461 tablet_copy_client.cc:523] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.845407 9664 raft_consensus.cc:1261] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.845480 7187 raft_consensus.cc:1261] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:53.845974 14441 consensus_queue.cc:1035] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.846272 14447 consensus_queue.cc:1035] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:53.847611 14461 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
W20240503 14:13:53.847836 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e14af3987fbc44e896615d5229d815d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.859261 14447 raft_consensus.cc:2932] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:53.860219 14461 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.860635 14461 tablet_bootstrap.cc:492] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:53.860945 14461 ts_tablet_manager.cc:1367] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20240503 14:13:53.861676 14461 raft_consensus.cc:348] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:53.861862 14461 raft_consensus.cc:729] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:53.862087 14461 consensus_queue.cc:260] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:53.862617 14461 ts_tablet_manager.cc:1398] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:53.863330 7190 raft_consensus.cc:2932] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:53.868992 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-b9a0d6f38293413aba812c473ab0973e received from {username='slave'} at 127.0.186.5:46745
I20240503 14:13:53.869154 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-b9a0d6f38293413aba812c473ab0973e on tablet b9a0d6f38293413aba812c473ab0973e with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:53.869746 9662 raft_consensus.cc:2932] T e14af3987fbc44e896615d5229d815d7 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:53.870357 6655 catalog_manager.cc:5484] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 4, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:53.883414 14475 ts_tablet_manager.cc:902] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:53.885501 14475 tablet_copy_client.cc:308] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:53.886313 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 5dc7cefd974149148dcf856a50ea168c from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:46691)
I20240503 14:13:53.886469 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 5dc7cefd974149148dcf856a50ea168c from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:46691: session id = 09723f236d8044538d920a37b5d9aa8c-5dc7cefd974149148dcf856a50ea168c
I20240503 14:13:53.889101 9819 tablet_copy_source_session.cc:205] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.890880 14475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dc7cefd974149148dcf856a50ea168c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.897488 14475 tablet_copy_client.cc:783] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.897732 14475 tablet_copy_client.cc:647] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.899168 14475 tablet_copy_client.cc:523] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.903375 14475 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:53.905397 9799 consensus_queue.cc:237] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:53.908077 6931 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:13:53.908587 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef4166613a824863b6c006dafb3783d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.908854 14045 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:53.908944 14475 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.909083 9532 raft_consensus.cc:1261] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:53.909312 14475 tablet_bootstrap.cc:492] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:53.909543 14475 ts_tablet_manager.cc:1367] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240503 14:13:53.909687 14297 consensus_queue.cc:1035] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:53.910126 14475 raft_consensus.cc:348] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:53.910287 14475 raft_consensus.cc:729] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:53.910460 14475 consensus_queue.cc:260] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:53.916119 14475 ts_tablet_manager.cc:1398] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:53.916851 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-5dc7cefd974149148dcf856a50ea168c received from {username='slave'} at 127.0.186.3:46691
I20240503 14:13:53.916985 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-5dc7cefd974149148dcf856a50ea168c on tablet 5dc7cefd974149148dcf856a50ea168c with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:53.916657 14297 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:53.921407 14488 ts_tablet_manager.cc:902] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:53.921775 9532 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:53.924448 6931 raft_consensus.cc:2932] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:53.934458 6652 catalog_manager.cc:5484] T ef4166613a824863b6c006dafb3783d3 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 4 to 5, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:53.940410 14488 tablet_copy_client.cc:308] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.6:33971
I20240503 14:13:53.941238 9685 tablet_copy_service.cc:140] P f03dc37fe3234033bc665bed4f1b51f5: Received BeginTabletCopySession request for tablet c3f2bd341493482f9519e6dd74cd2f3d from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:42691)
I20240503 14:13:53.941449 9685 tablet_copy_service.cc:161] P f03dc37fe3234033bc665bed4f1b51f5: Beginning new tablet copy session on tablet c3f2bd341493482f9519e6dd74cd2f3d from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:42691: session id = f1a669e28f014fe287932acf56bebc13-c3f2bd341493482f9519e6dd74cd2f3d
I20240503 14:13:53.944253 9685 tablet_copy_source_session.cc:205] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:53.950103 14488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3f2bd341493482f9519e6dd74cd2f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:53.959014 14488 tablet_copy_client.cc:783] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:53.959259 14488 tablet_copy_client.cc:647] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:53.961221 14488 tablet_copy_client.cc:523] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:53.961349 7187 consensus_queue.cc:237] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:53.963887 9799 raft_consensus.cc:1261] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:53.964084 9532 raft_consensus.cc:1261] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:53.964406 7125 consensus_peers.cc:485] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16b01e1bab7949cfb014c446c08434a8. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:53.964761 14065 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.965256 14488 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:53.965354 14065 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:53.974617 14342 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.975513 9799 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.977035 14488 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:53.977563 14488 tablet_bootstrap.cc:492] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:53.978063 14488 ts_tablet_manager.cc:1367] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240503 14:13:53.981173 14488 raft_consensus.cc:348] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:53.981607 14488 raft_consensus.cc:729] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:53.981496 9528 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:53.988422 14488 consensus_queue.cc:260] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:53.988584 6652 catalog_manager.cc:5484] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:53.989646 14488 ts_tablet_manager.cc:1398] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 14:13:53.991812 9685 tablet_copy_service.cc:342] P f03dc37fe3234033bc665bed4f1b51f5: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-c3f2bd341493482f9519e6dd74cd2f3d received from {username='slave'} at 127.0.186.1:42691
I20240503 14:13:53.992015 9685 tablet_copy_service.cc:433] P f03dc37fe3234033bc665bed4f1b51f5: ending tablet copy session f1a669e28f014fe287932acf56bebc13-c3f2bd341493482f9519e6dd74cd2f3d on tablet c3f2bd341493482f9519e6dd74cd2f3d with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:54.006032 14510 ts_tablet_manager.cc:902] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:54.016255 14510 tablet_copy_client.cc:308] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.3:44691
I20240503 14:13:54.017230 7081 tablet_copy_service.cc:140] P 09723f236d8044538d920a37b5d9aa8c: Received BeginTabletCopySession request for tablet 5895d998bfb3439386fcbc2c74f8d9a6 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:58913)
I20240503 14:13:54.017450 7081 tablet_copy_service.cc:161] P 09723f236d8044538d920a37b5d9aa8c: Beginning new tablet copy session on tablet 5895d998bfb3439386fcbc2c74f8d9a6 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:58913: session id = 3efdbd3506274dba846ae40404f0572c-5895d998bfb3439386fcbc2c74f8d9a6
I20240503 14:13:54.021370 7081 tablet_copy_source_session.cc:205] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.025276 14505 ts_tablet_manager.cc:902] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:54.025595 14510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5895d998bfb3439386fcbc2c74f8d9a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.034442 14510 tablet_copy_client.cc:783] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.034767 14510 tablet_copy_client.cc:647] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.038349 14510 tablet_copy_client.cc:523] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.041008 9662 consensus_queue.cc:237] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:54.043354 14510 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:54.050107 14505 tablet_copy_client.cc:308] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:54.050969 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 61b62d3aa0bc46c3be67eadc21466ec6 from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:51299)
I20240503 14:13:54.051132 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 61b62d3aa0bc46c3be67eadc21466ec6 from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:51299: session id = e1bf5fb7402c46829974e2f74cd88708-61b62d3aa0bc46c3be67eadc21466ec6
I20240503 14:13:54.053016 6931 raft_consensus.cc:1261] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:54.053334 9600 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 -> Peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): Couldn't send request to peer 3efdbd3506274dba846ae40404f0572c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f2f54c0c48b848ed898534efef55fb62. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.053887 6951 tablet_copy_source_session.cc:205] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.053820 14409 consensus_queue.cc:1035] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.053925 9928 raft_consensus.cc:1261] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.055197 14409 consensus_queue.cc:1035] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.055761 14510 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.056241 14510 tablet_bootstrap.cc:492] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:54.056521 14510 ts_tablet_manager.cc:1367] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240503 14:13:54.056584 14505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61b62d3aa0bc46c3be67eadc21466ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.057355 14510 raft_consensus.cc:348] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:54.057564 14510 raft_consensus.cc:729] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:54.057754 14510 consensus_queue.cc:260] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:54.058296 14510 ts_tablet_manager.cc:1398] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:13:54.058454 14409 raft_consensus.cc:2932] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:54.059320 7081 tablet_copy_service.cc:342] P 09723f236d8044538d920a37b5d9aa8c: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-5895d998bfb3439386fcbc2c74f8d9a6 received from {username='slave'} at 127.0.186.7:58913
I20240503 14:13:54.059430 7081 tablet_copy_service.cc:433] P 09723f236d8044538d920a37b5d9aa8c: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-5895d998bfb3439386fcbc2c74f8d9a6 on tablet 5895d998bfb3439386fcbc2c74f8d9a6 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:54.062858 9928 raft_consensus.cc:2932] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:54.065084 14505 tablet_copy_client.cc:783] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.065363 14505 tablet_copy_client.cc:647] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.065194 6931 raft_consensus.cc:2932] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } } }
I20240503 14:13:54.071460 6654 catalog_manager.cc:5484] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) added. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:54.083123 14505 tablet_copy_client.cc:523] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.086812 14505 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
I20240503 14:13:54.094031 14505 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.094352 14505 tablet_bootstrap.cc:492] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:54.094559 14505 ts_tablet_manager.cc:1367] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 14:13:54.095353 14505 raft_consensus.cc:348] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:54.095788 14505 raft_consensus.cc:729] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:54.096221 14505 consensus_queue.cc:260] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:54.097327 14505 ts_tablet_manager.cc:1398] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:54.099836 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-61b62d3aa0bc46c3be67eadc21466ec6 received from {username='slave'} at 127.0.186.4:51299
I20240503 14:13:54.100131 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-61b62d3aa0bc46c3be67eadc21466ec6 on tablet 61b62d3aa0bc46c3be67eadc21466ec6 with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:54.109305 14529 ts_tablet_manager.cc:902] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:54.109742 9664 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:54.121873 9799 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:13:54.122668 7187 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240503 14:13:54.123173 14409 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240503 14:13:54.123925 14410 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240503 14:13:54.124295 9601 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.129770 14529 tablet_copy_client.cc:308] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.2:45285
I20240503 14:13:54.133235 6951 tablet_copy_service.cc:140] P 7186ed8510424fd597173eaf10c57b12: Received BeginTabletCopySession request for tablet 933c08460b9745feab9bcb56c27bdbb9 from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:60021)
I20240503 14:13:54.133423 6951 tablet_copy_service.cc:161] P 7186ed8510424fd597173eaf10c57b12: Beginning new tablet copy session on tablet 933c08460b9745feab9bcb56c27bdbb9 from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:60021: session id = f03dc37fe3234033bc665bed4f1b51f5-933c08460b9745feab9bcb56c27bdbb9
I20240503 14:13:54.133996 14409 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:54.136538 6951 tablet_copy_source_session.cc:205] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.141881 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 933c08460b9745feab9bcb56c27bdbb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.163491 14529 tablet_copy_client.cc:783] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.164737 14529 tablet_copy_client.cc:647] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.167004 7187 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:54.170657 9799 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:54.171885 6654 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index 10 to 11, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 2 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:54.184278 14529 tablet_copy_client.cc:523] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.189006 14529 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:54.199271 14488 ts_tablet_manager.cc:902] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:54.209196 14529 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.209766 14529 tablet_bootstrap.cc:492] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:54.210055 14529 ts_tablet_manager.cc:1367] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.005s
I20240503 14:13:54.211094 14529 raft_consensus.cc:348] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:54.211457 14529 raft_consensus.cc:729] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:54.211747 14529 consensus_queue.cc:260] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:54.212842 14529 ts_tablet_manager.cc:1398] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 14:13:54.213757 6951 tablet_copy_service.cc:342] P 7186ed8510424fd597173eaf10c57b12: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-933c08460b9745feab9bcb56c27bdbb9 received from {username='slave'} at 127.0.186.6:60021
I20240503 14:13:54.213904 6951 tablet_copy_service.cc:433] P 7186ed8510424fd597173eaf10c57b12: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-933c08460b9745feab9bcb56c27bdbb9 on tablet 933c08460b9745feab9bcb56c27bdbb9 with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:54.243252 14549 ts_tablet_manager.cc:902] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:54.265466 14488 tablet_copy_client.cc:308] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:54.266340 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet f47877cdf5f04676861b3dc4708f3920 from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:40149)
I20240503 14:13:54.266541 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet f47877cdf5f04676861b3dc4708f3920 from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:40149: session id = f1a669e28f014fe287932acf56bebc13-f47877cdf5f04676861b3dc4708f3920
I20240503 14:13:54.268959 9950 tablet_copy_source_session.cc:205] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.270466 14488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f47877cdf5f04676861b3dc4708f3920. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.270830 9799 consensus_queue.cc:237] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:54.277732 14488 tablet_copy_client.cc:783] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.277992 14488 tablet_copy_client.cc:647] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.278610 9662 raft_consensus.cc:1261] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.278872 9528 raft_consensus.cc:1261] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:54.279039 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e5922e2c6624846b64abf6df41a5601. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.279690 14297 consensus_queue.cc:1035] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.280254 14297 consensus_queue.cc:1035] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.283851 14045 raft_consensus.cc:2932] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:54.287557 14549 tablet_copy_client.cc:308] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:34101
I20240503 14:13:54.288350 6820 tablet_copy_service.cc:140] P f1a669e28f014fe287932acf56bebc13: Received BeginTabletCopySession request for tablet d1df9d5b8a3d4a489cbccca67b223168 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:33179)
I20240503 14:13:54.288516 6820 tablet_copy_service.cc:161] P f1a669e28f014fe287932acf56bebc13: Beginning new tablet copy session on tablet d1df9d5b8a3d4a489cbccca67b223168 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:33179: session id = 7186ed8510424fd597173eaf10c57b12-d1df9d5b8a3d4a489cbccca67b223168
I20240503 14:13:54.289557 14488 tablet_copy_client.cc:523] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.291152 6820 tablet_copy_source_session.cc:205] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.294418 14488 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:54.294361 9662 raft_consensus.cc:2932] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:54.296625 14549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1df9d5b8a3d4a489cbccca67b223168. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.300467 9528 raft_consensus.cc:2932] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:54.301846 6655 catalog_manager.cc:5484] T 9e5922e2c6624846b64abf6df41a5601 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:54.309916 14549 tablet_copy_client.cc:783] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.310166 14549 tablet_copy_client.cc:647] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.312201 14549 tablet_copy_client.cc:523] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.315984 14488 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.316311 14549 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:54.316438 14488 tablet_bootstrap.cc:492] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:54.316673 14488 ts_tablet_manager.cc:1367] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.000s
I20240503 14:13:54.317410 14488 raft_consensus.cc:348] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:54.317669 14488 raft_consensus.cc:729] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:54.317879 14488 consensus_queue.cc:260] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:54.318578 14488 ts_tablet_manager.cc:1398] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 14:13:54.321466 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-f47877cdf5f04676861b3dc4708f3920 received from {username='slave'} at 127.0.186.1:40149
I20240503 14:13:54.321615 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session f1a669e28f014fe287932acf56bebc13-f47877cdf5f04676861b3dc4708f3920 on tablet f47877cdf5f04676861b3dc4708f3920 with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:54.327569 6800 raft_consensus.cc:1203] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.333963 14549 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.334391 14549 tablet_bootstrap.cc:492] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:54.334650 14549 ts_tablet_manager.cc:1367] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.004s
I20240503 14:13:54.335459 14549 raft_consensus.cc:348] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:54.335721 14549 raft_consensus.cc:729] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:54.335923 14549 consensus_queue.cc:260] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:54.338008 14549 ts_tablet_manager.cc:1398] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:54.338831 6820 tablet_copy_service.cc:342] P f1a669e28f014fe287932acf56bebc13: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-d1df9d5b8a3d4a489cbccca67b223168 received from {username='slave'} at 127.0.186.2:33179
I20240503 14:13:54.338975 6820 tablet_copy_service.cc:433] P f1a669e28f014fe287932acf56bebc13: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-d1df9d5b8a3d4a489cbccca67b223168 on tablet d1df9d5b8a3d4a489cbccca67b223168 with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:54.351378 9528 raft_consensus.cc:1203] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.361095 14529 ts_tablet_manager.cc:902] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:54.362565 14475 ts_tablet_manager.cc:902] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:54.367187 14529 tablet_copy_client.cc:308] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:54.367902 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet ef4166613a824863b6c006dafb3783d3 from peer f03dc37fe3234033bc665bed4f1b51f5 ({username='slave'} at 127.0.186.6:51981)
I20240503 14:13:54.368088 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet ef4166613a824863b6c006dafb3783d3 from peer f03dc37fe3234033bc665bed4f1b51f5 at {username='slave'} at 127.0.186.6:51981: session id = f03dc37fe3234033bc665bed4f1b51f5-ef4166613a824863b6c006dafb3783d3
I20240503 14:13:54.371104 14475 tablet_copy_client.cc:308] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.5:36609
I20240503 14:13:54.371263 9819 tablet_copy_source_session.cc:205] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.371765 9553 tablet_copy_service.cc:140] P 5b7e36bf354f480c90859cb8867b59cf: Received BeginTabletCopySession request for tablet e14af3987fbc44e896615d5229d815d7 from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:55131)
I20240503 14:13:54.371953 9553 tablet_copy_service.cc:161] P 5b7e36bf354f480c90859cb8867b59cf: Beginning new tablet copy session on tablet e14af3987fbc44e896615d5229d815d7 from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:55131: session id = 09723f236d8044538d920a37b5d9aa8c-e14af3987fbc44e896615d5229d815d7
I20240503 14:13:54.372896 14529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef4166613a824863b6c006dafb3783d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.374418 9553 tablet_copy_source_session.cc:205] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.375341 7187 consensus_queue.cc:237] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:54.375568 14475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14af3987fbc44e896615d5229d815d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.377902 6800 raft_consensus.cc:1261] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.378041 9799 raft_consensus.cc:1261] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.378794 14529 tablet_copy_client.cc:783] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.379068 14529 tablet_copy_client.cc:647] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.379014 14342 consensus_queue.cc:1035] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.379547 14565 consensus_queue.cc:1035] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:54.380398 7125 consensus_peers.cc:485] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 964b1598fb17446dacfe2b5931df5d41. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.382033 7187 raft_consensus.cc:1203] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.383059 9797 raft_consensus.cc:1203] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.383968 14529 tablet_copy_client.cc:523] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.390698 14529 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap starting.
I20240503 14:13:54.397850 14565 raft_consensus.cc:2932] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:54.400516 9799 raft_consensus.cc:2932] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:54.411463 6655 catalog_manager.cc:5484] T 964b1598fb17446dacfe2b5931df5d41 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:54.441926 7059 raft_consensus.cc:1203] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.442620 6800 raft_consensus.cc:2932] T 964b1598fb17446dacfe2b5931df5d41 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } } }
I20240503 14:13:54.449323 14475 tablet_copy_client.cc:783] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.449612 14475 tablet_copy_client.cc:647] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.452590 14475 tablet_copy_client.cc:523] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.456686 14529 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.457140 14529 tablet_bootstrap.cc:492] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: Bootstrap complete.
I20240503 14:13:54.457249 14475 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:54.457404 14529 ts_tablet_manager.cc:1367] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: Time spent bootstrapping tablet: real 0.067s user 0.021s sys 0.029s
I20240503 14:13:54.458307 14488 ts_tablet_manager.cc:902] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:54.458238 14529 raft_consensus.cc:348] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:54.458514 14529 raft_consensus.cc:729] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f03dc37fe3234033bc665bed4f1b51f5, State: Initialized, Role: LEARNER
I20240503 14:13:54.458742 14529 consensus_queue.cc:260] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: NON_VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: true } }
I20240503 14:13:54.460438 14529 ts_tablet_manager.cc:1398] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:54.461299 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session f03dc37fe3234033bc665bed4f1b51f5-ef4166613a824863b6c006dafb3783d3 received from {username='slave'} at 127.0.186.6:51981
I20240503 14:13:54.461474 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session f03dc37fe3234033bc665bed4f1b51f5-ef4166613a824863b6c006dafb3783d3 on tablet ef4166613a824863b6c006dafb3783d3 with peer f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:54.478585 14475 tablet.cc:1722] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 14:13:54.479202 14475 tablet.cc:1620] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:54.479314 14475 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20240503 14:13:54.480370 14475 tablet.cc:1722] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 14:13:54.480767 14475 tablet.cc:1620] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: MemRowSet was empty: no flush needed.
I20240503 14:13:54.480852 14475 tablet_metadata.cc:623] Not flushing: waiting for 1 pins to be released.
I20240503 14:13:54.481786 14475 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.482182 14475 tablet_bootstrap.cc:492] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:54.485980 14475 ts_tablet_manager.cc:1367] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.029s user 0.014s sys 0.005s
I20240503 14:13:54.486711 14475 raft_consensus.cc:348] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:54.486954 14475 raft_consensus.cc:729] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:54.487133 14475 consensus_queue.cc:260] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:54.487884 14488 tablet_copy_client.cc:308] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:54.489091 14475 ts_tablet_manager.cc:1398] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240503 14:13:54.489743 9553 tablet_copy_service.cc:342] P 5b7e36bf354f480c90859cb8867b59cf: Request end of tablet copy session 09723f236d8044538d920a37b5d9aa8c-e14af3987fbc44e896615d5229d815d7 received from {username='slave'} at 127.0.186.3:55131
I20240503 14:13:54.489871 9553 tablet_copy_service.cc:433] P 5b7e36bf354f480c90859cb8867b59cf: ending tablet copy session 09723f236d8044538d920a37b5d9aa8c-e14af3987fbc44e896615d5229d815d7 on tablet e14af3987fbc44e896615d5229d815d7 with peer 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:54.494974 7187 consensus_queue.cc:237] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:54.497939 7059 raft_consensus.cc:1261] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.497972 6931 raft_consensus.cc:1261] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.498127 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 16b01e1bab7949cfb014c446c08434a8 from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:54537)
I20240503 14:13:54.498253 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 16b01e1bab7949cfb014c446c08434a8 from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:54537: session id = f1a669e28f014fe287932acf56bebc13-16b01e1bab7949cfb014c446c08434a8
W20240503 14:13:54.498435 7125 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101): Couldn't send request to peer f1a669e28f014fe287932acf56bebc13. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1708b19d3c5c4d308f4358bd42182c80. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.498539 14342 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.498883 14065 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.500758 7211 tablet_copy_source_session.cc:205] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.501772 14342 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:54.502298 7059 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:54.506376 6931 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } }
I20240503 14:13:54.507711 6652 catalog_manager.cc:5484] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index -1 to 2, NON_VOTER f1a669e28f014fe287932acf56bebc13 (127.0.186.1) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:54.511042 14488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16b01e1bab7949cfb014c446c08434a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.524266 14488 tablet_copy_client.cc:783] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.524529 14488 tablet_copy_client.cc:647] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.530658 14488 tablet_copy_client.cc:523] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.533703 14488 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:54.539734 14488 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.540412 14488 tablet_bootstrap.cc:492] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:54.540663 14488 ts_tablet_manager.cc:1367] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240503 14:13:54.541324 14488 raft_consensus.cc:348] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:54.541560 14488 raft_consensus.cc:729] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
I20240503 14:13:54.541746 14488 consensus_queue.cc:260] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:54.542222 14488 ts_tablet_manager.cc:1398] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240503 14:13:54.543936 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-16b01e1bab7949cfb014c446c08434a8 received from {username='slave'} at 127.0.186.1:54537
I20240503 14:13:54.544076 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session f1a669e28f014fe287932acf56bebc13-16b01e1bab7949cfb014c446c08434a8 on tablet 16b01e1bab7949cfb014c446c08434a8 with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:54.553735 9928 consensus_queue.cc:237] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:54.556239 6800 raft_consensus.cc:1261] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.556566 7059 raft_consensus.cc:1261] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.557065 14354 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.557397 14446 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:54.557897 9864 consensus_peers.cc:485] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc -> Peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337): Couldn't send request to peer e1bf5fb7402c46829974e2f74cd88708. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 92b477db785d47749fabf31ef385e9d2. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.564337 14354 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:54.565147 7059 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:54.566797 6800 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } } }
I20240503 14:13:54.571401 6655 catalog_manager.cc:5484] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:54.580257 9664 raft_consensus.cc:1203] T 933c08460b9745feab9bcb56c27bdbb9 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.587152 6798 raft_consensus.cc:1203] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.615451 6798 consensus_queue.cc:237] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:54.617905 9797 raft_consensus.cc:1261] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:54.618270 6736 consensus_peers.cc:485] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 -> Peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): Couldn't send request to peer 5b7e36bf354f480c90859cb8867b59cf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00ecf4e868774482b4600566943fd224. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:54.618597 14378 consensus_queue.cc:1035] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.618893 9664 raft_consensus.cc:1261] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 14:13:54.619678 14378 consensus_queue.cc:1035] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:54.620651 14612 ts_tablet_manager.cc:902] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: Initiating tablet copy from peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:54.632656 14457 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:54.634698 9661 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:54.636165 14475 ts_tablet_manager.cc:902] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:54.643500 6655 catalog_manager.cc:5484] T 00ecf4e868774482b4600566943fd224 P f03dc37fe3234033bc665bed4f1b51f5 reported cstate change: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:54.651420 14612 tablet_copy_client.cc:308] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.6:33971
I20240503 14:13:54.652210 9685 tablet_copy_service.cc:140] P f03dc37fe3234033bc665bed4f1b51f5: Received BeginTabletCopySession request for tablet f2f54c0c48b848ed898534efef55fb62 from peer 3efdbd3506274dba846ae40404f0572c ({username='slave'} at 127.0.186.7:40383)
I20240503 14:13:54.652437 9685 tablet_copy_service.cc:161] P f03dc37fe3234033bc665bed4f1b51f5: Beginning new tablet copy session on tablet f2f54c0c48b848ed898534efef55fb62 from peer 3efdbd3506274dba846ae40404f0572c at {username='slave'} at 127.0.186.7:40383: session id = 3efdbd3506274dba846ae40404f0572c-f2f54c0c48b848ed898534efef55fb62
I20240503 14:13:54.655823 9685 tablet_copy_source_session.cc:205] T f2f54c0c48b848ed898534efef55fb62 P f03dc37fe3234033bc665bed4f1b51f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.657364 14612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2f54c0c48b848ed898534efef55fb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.661926 9797 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } } }
I20240503 14:13:54.668941 14612 tablet_copy_client.cc:783] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.669281 14612 tablet_copy_client.cc:647] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:54.671767 14612 tablet_copy_client.cc:523] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.678534 14612 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: Bootstrap starting.
I20240503 14:13:54.690145 6928 raft_consensus.cc:1203] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.716447 14612 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:54.718071 14612 tablet_bootstrap.cc:492] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: Bootstrap complete.
I20240503 14:13:54.721040 14612 ts_tablet_manager.cc:1367] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.003s
I20240503 14:13:54.722301 14612 raft_consensus.cc:348] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:54.722649 14612 raft_consensus.cc:729] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Initialized, Role: LEARNER
I20240503 14:13:54.723096 14612 consensus_queue.cc:260] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:54.724404 14612 ts_tablet_manager.cc:1398] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 14:13:54.725186 9685 tablet_copy_service.cc:342] P f03dc37fe3234033bc665bed4f1b51f5: Request end of tablet copy session 3efdbd3506274dba846ae40404f0572c-f2f54c0c48b848ed898534efef55fb62 received from {username='slave'} at 127.0.186.7:40383
I20240503 14:13:54.725324 9685 tablet_copy_service.cc:433] P f03dc37fe3234033bc665bed4f1b51f5: ending tablet copy session 3efdbd3506274dba846ae40404f0572c-f2f54c0c48b848ed898534efef55fb62 on tablet f2f54c0c48b848ed898534efef55fb62 with peer 3efdbd3506274dba846ae40404f0572c
I20240503 14:13:54.742056 14475 tablet_copy_client.cc:308] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.6:33971
I20240503 14:13:54.743398 9685 tablet_copy_service.cc:140] P f03dc37fe3234033bc665bed4f1b51f5: Received BeginTabletCopySession request for tablet 9b154f780fc742fa9b0efd609c307f95 from peer 09723f236d8044538d920a37b5d9aa8c ({username='slave'} at 127.0.186.3:35985)
I20240503 14:13:54.743563 9685 tablet_copy_service.cc:161] P f03dc37fe3234033bc665bed4f1b51f5: Beginning new tablet copy session on tablet 9b154f780fc742fa9b0efd609c307f95 from peer 09723f236d8044538d920a37b5d9aa8c at {username='slave'} at 127.0.186.3:35985: session id = 09723f236d8044538d920a37b5d9aa8c-9b154f780fc742fa9b0efd609c307f95
I20240503 14:13:54.747478 9685 tablet_copy_source_session.cc:205] T 9b154f780fc742fa9b0efd609c307f95 P f03dc37fe3234033bc665bed4f1b51f5: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:54.750049 14475 ts_tablet_manager.cc:1875] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240503 14:13:54.761279 14475 ts_tablet_manager.cc:1888] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240503 14:13:54.762616 14475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b154f780fc742fa9b0efd609c307f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:54.765009 14379 raft_consensus.cc:1050] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:54.765830 14379 consensus_queue.cc:237] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:54.778194 9532 raft_consensus.cc:1261] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.778812 14071 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:54.785511 14529 ts_tablet_manager.cc:902] T 964b1598fb17446dacfe2b5931df5d41 P f03dc37fe3234033bc665bed4f1b51f5: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
F20240503 14:13:54.785892 14529 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1d3ec4f0833d4d489f774757312efb32.queue_time_us, tablet-download-pool-1d3ec4f0833d4d489f774757312efb32.queue_time_us) (tablet-download-pool-1d3ec4f0833d4d489f774757312efb32.run_cpu_time_us, tablet-download-pool-1d3ec4f0833d4d489f774757312efb32.run_cpu_time_us) (tablet-download-pool-1d3ec4f0833d4d489f774757312efb32.run_wall_time_us, tablet-download-pool-1d3ec4f0833d4d489f774757312efb32.run_wall_time_us) (tablet-download-pool-20c4e7c22c904daebe97897320f6226e.queue_time_us, tablet-download-pool-20c4e7c22c904daebe97897320f6226e.queue_time_us) (tablet-download-pool-20c4e7c22c904daebe97897320f6226e.run_cpu_time_us, tablet-download-pool-20c4e7c22c904daebe97897320f6226e.run_cpu_time_us) (tablet-download-pool-20c4e7c22c904daebe97897320f6226e.run_wall_time_us, tablet-download-pool-20c4e7c22c904daebe97897320f6226e.run_wall_time_us) (tablet-download-pool-2d5781b2310f4814aaebdcb715b7ddac.queue_time_us, tablet-download-pool-2d5781b2310f4814aaebdcb715b7ddac.queue_time_us) (tablet-download-pool-2d5781b2310f4814aaebdcb715b7ddac.run_cpu_time_us, tablet-download-pool-2d5781b2310f4814aaebdcb715b7ddac.run_cpu_time_us) (tablet-download-pool-2d5781b2310f4814aaebdcb715b7ddac.run_wall_time_us, tablet-download-pool-2d5781b2310f4814aaebdcb715b7ddac.run_wall_time_us) (tablet-download-pool-669d722d3eb1475dba616ed72cb775ac.queue_time_us, tablet-download-pool-669d722d3eb1475dba616ed72cb775ac.queue_time_us) (tablet-download-pool-669d722d3eb1475dba616ed72cb775ac.run_cpu_time_us, tablet-download-pool-669d722d3eb1475dba616ed72cb775ac.run_cpu_time_us) (tablet-download-pool-669d722d3eb1475dba616ed72cb775ac.run_wall_time_us, tablet-download-pool-669d722d3eb1475dba616ed72cb775ac.run_wall_time_us) (tablet-download-pool-6e0bb82edac2464b9d618c1c80807fb2.queue_time_us, tablet-download-pool-6e0bb82edac2464b9d618c1c80807fb2.queue_time_us) (tablet-download-pool-6e0bb82edac2464b9d618c1c80807fb2.run_cpu_time_us, tablet-download-pool-6e0bb82edac2464b9d618c1c80807fb2.run_cpu_time_us) (tablet-download-pool-6e0bb82edac2464b9d618c1c80807fb2.run_wall_time_us, tablet-download-pool-6e0bb82edac2464b9d618c1c80807fb2.run_wall_time_us) (tablet-download-pool-77d3f36de08d4f1986e42e86e1b95fbf.queue_time_us, tablet-download-pool-77d3f36de08d4f1986e42e86e1b95fbf.queue_time_us) (tablet-download-pool-77d3f36de08d4f1986e42e86e1b95fbf.run_cpu_time_us, tablet-download-pool-77d3f36de08d4f1986e42e86e1b95fbf.run_cpu_time_us) (tablet-download-pool-77d3f36de08d4f1986e42e86e1b95fbf.run_wall_time_us, tablet-download-pool-77d3f36de08d4f1986e42e86e1b95fbf.run_wall_time_us) (tablet-download-pool-933c08460b9745feab9bcb56c27bdbb9.queue_time_us, tablet-download-pool-933c08460b9745feab9bcb56c27bdbb9.queue_time_us) (tablet-download-pool-933c08460b9745feab9bcb56c27bdbb9.run_cpu_time_us, tablet-download-pool-933c08460b9745feab9bcb56c27bdbb9.run_cpu_time_us) (tablet-download-pool-933c08460b9745feab9bcb56c27bdbb9.run_wall_time_us, tablet-download-pool-933c08460b9745feab9bcb56c27bdbb9.run_wall_time_us) (tablet-download-pool-964b1598fb17446dacfe2b5931df5d41.queue_time_us, tablet-download-pool-964b1598fb17446dacfe2b5931df5d41.queue_time_us) (tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.queue_time_us, tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.queue_time_us) (tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.run_cpu_time_us, tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.run_cpu_time_us) (tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.run_wall_time_us, tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.run_wall_time_us) (tablet-download-pool-9f30640fc81e4aeead38e3f8659c056a.queue_time_us, tablet-download-pool-9f30640fc81e4aeead38e3f8659c056a.queue_time_us) (tablet-download-pool-9f30640fc81e4aeead38e3f8659c056a.run_cpu_time_us, tablet-download-pool-9f30640fc81e4aeead38e3f8659c056a.run_cpu_time_us) (tablet-download-pool-9f30640fc81e4aeead38e3f8659c056a.run_wall_time_us, tablet-download-pool-9f30640fc81e4aeead38e3f8659c056a.run_wall_time_us) (tablet-download-pool-a44ed92d65114bfda2e0f38054b17ada.queue_time_us, tablet-download-pool-a44ed92d65114bfda2e0f38054b17ada.queue_time_us) (tablet-download-pool-a44ed92d65114bfda2e0f38054b17ada.run_cpu_time_us, tablet-download-pool-a44ed92d65114bfda2e0f38054b17ada.run_cpu_time_us) (tablet-download-pool-a44ed92d65114bfda2e0f38054b17ada.run_wall_time_us, tablet-download-pool-a44ed92d65114bfda2e0f38054b17ada.run_wall_time_us) (tablet-download-pool-b3e2d7664e6c45d69f4f8b20dcf0e4d0.queue_time_us, tablet-download-pool-b3e2d7664e6c45d69f4f8b20dcf0e4d0.queue_time_us) (tablet-download-pool-b3e2d7664e6c45d69f4f8b20dcf0e4d0.run_cpu_time_us, tablet-download-pool-b3e2d7664e6c45d69f4f8b20dcf0e4d0.run_cpu_time_us) (tablet-download-pool-b3e2d7664e6c45d69f4f8b20dcf0e4d0.run_wall_time_us, tablet-download-pool-b3e2d7664e6c45d69f4f8b20dcf0e4d0.run_wall_time_us) (tablet-download-pool-b8014635e8294062b1c1985a100307ff.queue_time_us, tablet-download-pool-b8014635e8294062b1c1985a100307ff.queue_time_us) (tablet-download-pool-b8014635e8294062b1c1985a100307ff.run_cpu_time_us, tablet-download-pool-b8014635e8294062b1c1985a100307ff.run_cpu_time_us) (tablet-download-pool-b8014635e8294062b1c1985a100307ff.run_wall_time_us, tablet-download-pool-b8014635e8294062b1c1985a100307ff.run_wall_time_us) (tablet-download-pool-ef4166613a824863b6c006dafb3783d3.queue_time_us, tablet-download-pool-ef4166613a824863b6c006dafb3783d3.queue_time_us) (tablet-download-pool-ef4166613a824863b6c006dafb3783d3.run_cpu_time_us, tablet-download-pool-ef4166613a824863b6c006dafb3783d3.run_cpu_time_us) (tablet-download-pool-ef4166613a824863b6c006dafb3783d3.run_wall_time_us, tablet-download-pool-ef4166613a824863b6c006dafb3783d3.run_wall_time_us) (tablet-download-pool-f5738d4265f941519b90b07334d06af9.queue_time_us, tablet-download-pool-f5738d4265f941519b90b07334d06af9.queue_time_us) (tablet-download-pool-f5738d4265f941519b90b07334d06af9.run_cpu_time_us, tablet-download-pool-f5738d4265f941519b90b07334d06af9.run_cpu_time_us) (tablet-download-pool-f5738d4265f941519b90b07334d06af9.run_wall_time_us, tablet-download-pool-f5738d4265f941519b90b07334d06af9.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 14:13:54.786114 7059 raft_consensus.cc:1261] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
*** Check failure stack trace: ***
I20240503 14:13:54.786862 14071 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7faba1ce9dcd google::LogMessage::Fail() at ??:0
@ 0x7faba1cedb93 google::LogMessage::SendToLog() at ??:0
I20240503 14:13:54.794071 14409 raft_consensus.cc:1050] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:54.794979 14409 consensus_queue.cc:237] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:54.796626 9928 raft_consensus.cc:1261] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.797204 14071 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7faba1ce97cc google::LogMessage::Flush() at ??:0
@ 0x7faba1ceaf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 14:13:54.814311 6800 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.815066 14538 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:54.815296 9929 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.815932 14538 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:54.816833 9532 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f03dc37fe3234033bc665bed4f1b51f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.820137 9928 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:54.820505 14409 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:54.823984 14379 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
@ 0x7faba4a2a742 kudu::TraceMetrics::InternName() at ??:0
I20240503 14:13:54.834864 9532 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:54.842926 6655 catalog_manager.cc:5484] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:54.848846 14409 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
@ 0x7faba49f6f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 14:13:54.851909 7059 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:54.853276 14475 tablet_copy_client.cc:783] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:54.853780 14475 tablet_copy_client.cc:647] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Starting download of 1 WAL segments...
@ 0x7faba49f0cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 14:13:54.860113 6800 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:54.861541 9528 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:54.867960 9662 raft_consensus.cc:1203] T ef4166613a824863b6c006dafb3783d3 P f03dc37fe3234033bc665bed4f1b51f5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 14:13:54.868310 6928 consensus_queue.cc:237] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:54.871006 9929 raft_consensus.cc:1261] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:54.873656 14475 tablet_copy_client.cc:523] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:54.874145 14389 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:54.881090 9532 raft_consensus.cc:1261] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:54.882673 14379 consensus_queue.cc:1035] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:54.882949 6651 catalog_manager.cc:5484] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f03dc37fe3234033bc665bed4f1b51f5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:54.885037 14071 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:54.887866 14475 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap starting.
I20240503 14:13:54.890097 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b9a0d6f38293413aba812c473ab0973e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:54.889860 9929 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:54.891754 9928 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:54.894289 9529 raft_consensus.cc:2932] T b9a0d6f38293413aba812c473ab0973e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:54.899662 14377 raft_consensus.cc:1050] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:54.900539 14377 consensus_queue.cc:237] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:54.899288 6654 catalog_manager.cc:5484] T b9a0d6f38293413aba812c473ab0973e P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:54.909081 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet b9a0d6f38293413aba812c473ab0973e with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:54.926193 9532 raft_consensus.cc:1261] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.926508 14640 ts_tablet_manager.cc:902] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:54.926769 14114 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7fabb3e448de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 14:13:54.932636 9797 raft_consensus.cc:1261] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.933210 14114 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7fabb3e5cdaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 14:13:54.958331 6798 raft_consensus.cc:1203] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:54.967446 14488 ts_tablet_manager.cc:902] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
@ 0x7fabb3f87480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 14:13:54.975060 6931 raft_consensus.cc:1261] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.975724 14114 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:54.979724 14643 tablet_replica.cc:330] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:54.979959 14643 raft_consensus.cc:2229] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:54.980168 14643 raft_consensus.cc:2258] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:54.981981 14071 raft_consensus.cc:1050] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
I20240503 14:13:54.982825 14071 consensus_queue.cc:237] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:54.982981 7056 raft_consensus.cc:1203] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 14:13:54.986384 14643 ts_tablet_manager.cc:1875] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:13:54.992180 9865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 64670us
I20240503 14:13:54.995318 14653 ts_tablet_manager.cc:902] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:54.997617 7190 raft_consensus.cc:1261] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:54.998229 14379 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.000674 9928 raft_consensus.cc:1261] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.001278 14071 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.003360 14045 raft_consensus.cc:1050] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:55.004488 14045 consensus_queue.cc:237] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
@ 0x7fabb3fbfa8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 14:13:55.000322 9532 raft_consensus.cc:1261] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.017971 14071 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.026140 7056 raft_consensus.cc:1261] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.027189 13127 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.037405 9929 raft_consensus.cc:1261] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.038165 13127 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7fabb3fa2981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 14:13:55.053786 14114 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
@ 0x7fabb3fa270e std::_Function_handler<>::_M_invoke() at ??:0
I20240503 14:13:55.056360 9532 raft_consensus.cc:1261] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 14:13:55.059615 14379 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:55.061604 7187 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
W20240503 14:13:55.064961 9469 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 79175.2us
I20240503 14:13:55.066376 9929 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:55.068058 6651 catalog_manager.cc:5484] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 2 to 3, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:55.070887 6655 catalog_manager.cc:5484] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:55.072549 7059 consensus_queue.cc:237] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:55.077227 9532 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:55.078956 13127 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.079411 9799 raft_consensus.cc:1261] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.080448 14377 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.081694 6928 consensus_queue.cc:237] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:55.081547 9799 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:55.084935 14670 ts_tablet_manager.cc:902] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:55.085778 9929 raft_consensus.cc:1261] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.087092 14379 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.088133 14640 tablet_copy_client.cc:308] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:55.088897 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet 9e5922e2c6624846b64abf6df41a5601 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:50341)
I20240503 14:13:55.089079 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet 9e5922e2c6624846b64abf6df41a5601 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:50341: session id = 7186ed8510424fd597173eaf10c57b12-9e5922e2c6624846b64abf6df41a5601
I20240503 14:13:55.089547 9529 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:55.090888 7190 raft_consensus.cc:1261] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.091910 14379 consensus_queue.cc:1035] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.095047 14071 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:55.095793 9819 tablet_copy_source_session.cc:205] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.095518 7190 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:55.099570 14640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e5922e2c6624846b64abf6df41a5601. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.099947 14576 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:55.103452 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61b62d3aa0bc46c3be67eadc21466ec6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.104454 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5895d998bfb3439386fcbc2c74f8d9a6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.105906 9799 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:55.110240 6655 catalog_manager.cc:5484] T 61b62d3aa0bc46c3be67eadc21466ec6 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:55.113178 13127 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:55.113966 14640 tablet_copy_client.cc:783] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:55.114271 14640 tablet_copy_client.cc:647] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
@ 0x7faba49fab75 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 14:13:55.117002 9531 raft_consensus.cc:1261] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.118546 7059 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
@ 0x7faba49fe19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240503 14:13:55.119724 14640 tablet_copy_client.cc:523] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:55.121042 9929 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
@ 0x7faba49fe019 std::_Function_handler<>::_M_invoke() at ??:0
I20240503 14:13:55.124859 14640 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 14:13:55.129312 6658 catalog_manager.cc:5484] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:55.130143 14620 raft_consensus.cc:1050] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:55.130946 14620 consensus_queue.cc:237] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:55.137473 14645 raft_consensus.cc:1050] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:55.138059 6653 catalog_manager.cc:5484] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 4, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:55.138417 14645 consensus_queue.cc:237] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:55.149531 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 5895d998bfb3439386fcbc2c74f8d9a6 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:55.149660 14114 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.155826 6930 raft_consensus.cc:1261] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.156567 14378 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.156788 9929 raft_consensus.cc:1261] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.157405 14378 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7faba49d901e kudu::Thread::SuperviseThread() at ??:0
I20240503 14:13:55.175740 14389 raft_consensus.cc:1050] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER f03dc37fe3234033bc665bed4f1b51f5 to VOTER
I20240503 14:13:55.176823 14389 consensus_queue.cc:237] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:55.187011 14640 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:55.187562 14640 tablet_bootstrap.cc:492] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:55.187963 14640 ts_tablet_manager.cc:1367] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.001s
I20240503 14:13:55.189587 14640 raft_consensus.cc:348] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:55.189821 14640 raft_consensus.cc:729] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:55.190018 14640 consensus_queue.cc:260] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
@ 0x7faba7d5e6db start_thread at ??:0
@ 0x7faba0b9c71f clone at ??:0
I20240503 14:13:55.263921 9929 raft_consensus.cc:2932] T 61b62d3aa0bc46c3be67eadc21466ec6 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:55.191207 14640 ts_tablet_manager.cc:1398] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 14:13:55.197322 6928 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:55.170130 6931 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:55.272743 6930 raft_consensus.cc:1203] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:55.312314 14684 tablet_replica.cc:330] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:55.312666 14684 raft_consensus.cc:2229] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.312988 14684 raft_consensus.cc:2258] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.315778 14684 ts_tablet_manager.cc:1875] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.324326 6652 catalog_manager.cc:5484] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:55.216318 7059 raft_consensus.cc:1261] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.192456 9796 consensus_queue.cc:237] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:55.240125 9799 raft_consensus.cc:1261] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.239869 7056 raft_consensus.cc:1261] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.288853 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-9e5922e2c6624846b64abf6df41a5601 received from {username='slave'} at 127.0.186.2:50341
I20240503 14:13:55.348778 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-9e5922e2c6624846b64abf6df41a5601 on tablet 9e5922e2c6624846b64abf6df41a5601 with peer 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:55.350406 14071 consensus_queue.cc:1035] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.351763 14354 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.352437 14354 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.355429 7056 raft_consensus.cc:1261] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:55.247202 14475 log.cc:924] Time spent T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Append to log took a long time: real 0.231s user 0.001s sys 0.000s
I20240503 14:13:55.195696 14624 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:55.374846 14297 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.220232 6798 raft_consensus.cc:1261] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.233065 6800 raft_consensus.cc:1261] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.243974 14488 tablet_copy_client.cc:308] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.4:44337
I20240503 14:13:55.383704 14071 consensus_queue.cc:1035] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.384831 14354 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.393381 14045 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:55.393985 7059 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:55.399725 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5dc7cefd974149148dcf856a50ea168c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:55.407681 6799 raft_consensus.cc:2932] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:55.328984 14643 ts_tablet_manager.cc:1888] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:55.447700 14643 log.cc:1196] T b9a0d6f38293413aba812c473ab0973e P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/b9a0d6f38293413aba812c473ab0973e
I20240503 14:13:55.447342 7056 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:55.405728 14389 raft_consensus.cc:2932] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:55.424803 6800 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:55.438473 14684 ts_tablet_manager.cc:1888] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:55.468735 14684 log.cc:1196] T 5895d998bfb3439386fcbc2c74f8d9a6 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/5895d998bfb3439386fcbc2c74f8d9a6
I20240503 14:13:55.468710 7061 raft_consensus.cc:2932] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } }
I20240503 14:13:55.588538 14475 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:55.589555 14475 tablet_bootstrap.cc:492] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Bootstrap complete.
I20240503 14:13:55.590533 14475 ts_tablet_manager.cc:1367] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Time spent bootstrapping tablet: real 0.703s user 0.071s sys 0.009s
I20240503 14:13:55.401942 6652 catalog_manager.cc:5484] T 5dc7cefd974149148dcf856a50ea168c P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:55.327522 14653 tablet_copy_client.cc:308] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:55.383257 7211 tablet_copy_service.cc:140] P e1bf5fb7402c46829974e2f74cd88708: Received BeginTabletCopySession request for tablet 1708b19d3c5c4d308f4358bd42182c80 from peer f1a669e28f014fe287932acf56bebc13 ({username='slave'} at 127.0.186.1:54537)
I20240503 14:13:55.599836 7211 tablet_copy_service.cc:161] P e1bf5fb7402c46829974e2f74cd88708: Beginning new tablet copy session on tablet 1708b19d3c5c4d308f4358bd42182c80 from peer f1a669e28f014fe287932acf56bebc13 at {username='slave'} at 127.0.186.1:54537: session id = f1a669e28f014fe287932acf56bebc13-1708b19d3c5c4d308f4358bd42182c80
I20240503 14:13:55.404855 14565 raft_consensus.cc:1050] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:55.395684 9929 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:55.600937 14565 consensus_queue.cc:237] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:55.200404 9528 raft_consensus.cc:1261] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.448498 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet b9a0d6f38293413aba812c473ab0973e (table rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]) successfully deleted
W20240503 14:13:55.604055 6646 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 155527us
I20240503 14:13:55.605069 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet 92b477db785d47749fabf31ef385e9d2 from peer e1bf5fb7402c46829974e2f74cd88708 ({username='slave'} at 127.0.186.4:48587)
I20240503 14:13:55.605232 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet 92b477db785d47749fabf31ef385e9d2 from peer e1bf5fb7402c46829974e2f74cd88708 at {username='slave'} at 127.0.186.4:48587: session id = e1bf5fb7402c46829974e2f74cd88708-92b477db785d47749fabf31ef385e9d2
I20240503 14:13:55.423462 14446 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:55.608266 9950 tablet_copy_source_session.cc:205] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.610210 14653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92b477db785d47749fabf31ef385e9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.201973 9529 raft_consensus.cc:2932] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:55.602746 7211 tablet_copy_source_session.cc:205] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.627372 14488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1708b19d3c5c4d308f4358bd42182c80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.638080 14653 tablet_copy_client.cc:783] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:55.472386 6654 catalog_manager.cc:5484] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 2 to 3, f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:55.591468 14475 raft_consensus.cc:348] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:55.701753 14475 raft_consensus.cc:729] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Initialized, Role: LEARNER
I20240503 14:13:55.500284 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 5895d998bfb3439386fcbc2c74f8d9a6 (table rebalancer_extra_table_36 [id=fa04960f126f4afab9b4aac9e55289a1]) successfully deleted
I20240503 14:13:55.702083 14475 consensus_queue.cc:260] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
W20240503 14:13:55.702461 6644 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.186.2:45285, user_credentials={real_user=slave}} blocked reactor thread for 202174us
I20240503 14:13:55.702934 14475 ts_tablet_manager.cc:1398] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: Time spent starting tablet: real 0.112s user 0.002s sys 0.000s
I20240503 14:13:55.704720 6655 catalog_manager.cc:5484] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:55.705775 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet 5dc7cefd974149148dcf856a50ea168c with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:55.206912 9531 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:55.711079 9929 consensus_queue.cc:237] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:55.217218 9513 tablet_service.cc:1514] Processing DeleteTablet for tablet 61b62d3aa0bc46c3be67eadc21466ec6 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:33196
I20240503 14:13:55.628579 14620 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.769531 7061 raft_consensus.cc:1261] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.638345 14653 tablet_copy_client.cc:647] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:55.770193 14731 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.238561 14670 tablet_copy_client.cc:308] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.1:34101
I20240503 14:13:55.646361 14488 tablet_copy_client.cc:783] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:55.770990 14488 tablet_copy_client.cc:647] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:55.668282 6800 raft_consensus.cc:1261] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.578231 14660 raft_consensus.cc:1050] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:55.773591 6820 tablet_copy_service.cc:140] P f1a669e28f014fe287932acf56bebc13: Received BeginTabletCopySession request for tablet 00ecf4e868774482b4600566943fd224 from peer 5b7e36bf354f480c90859cb8867b59cf ({username='slave'} at 127.0.186.5:35165)
I20240503 14:13:55.773435 14660 consensus_queue.cc:237] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:55.773762 6820 tablet_copy_service.cc:161] P f1a669e28f014fe287932acf56bebc13: Beginning new tablet copy session on tablet 00ecf4e868774482b4600566943fd224 from peer 5b7e36bf354f480c90859cb8867b59cf at {username='slave'} at 127.0.186.5:35165: session id = 5b7e36bf354f480c90859cb8867b59cf-00ecf4e868774482b4600566943fd224
I20240503 14:13:55.670779 9529 raft_consensus.cc:1261] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.775606 14565 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.784520 7061 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:55.783725 14653 tablet_copy_client.cc:523] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:55.782698 14731 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:55.800770 14737 tablet_replica.cc:330] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:55.872313 14737 raft_consensus.cc:2229] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.872603 14737 raft_consensus.cc:2258] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.830768 7056 raft_consensus.cc:1261] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c [term 1 LEARNER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.785163 14565 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.876205 14737 ts_tablet_manager.cc:1875] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.836184 7187 raft_consensus.cc:1261] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:55.782070 9532 raft_consensus.cc:1261] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.801448 9528 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:55.835203 14738 tablet_replica.cc:330] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:55.883106 14737 ts_tablet_manager.cc:1888] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:55.883242 14737 log.cc:1196] T 5dc7cefd974149148dcf856a50ea168c P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/5dc7cefd974149148dcf856a50ea168c
I20240503 14:13:55.883195 14738 raft_consensus.cc:2229] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.883517 14738 raft_consensus.cc:2258] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.887221 14738 ts_tablet_manager.cc:1875] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:55.670087 9796 raft_consensus.cc:1261] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:55.677939 9799 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:55.843945 14045 raft_consensus.cc:1050] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:55.889250 14723 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:55.889252 14045 consensus_queue.cc:237] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:55.777243 6820 tablet_copy_source_session.cc:205] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:55.784160 14488 tablet_copy_client.cc:523] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:55.894273 14488 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: Bootstrap starting.
I20240503 14:13:55.769106 6798 raft_consensus.cc:1261] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:55.896464 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet 5dc7cefd974149148dcf856a50ea168c (table rebalancer_extra_table_42 [id=9befaf7f412c4b76b00af601821fa133]) successfully deleted
I20240503 14:13:55.899441 14645 consensus_queue.cc:1035] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.899634 14689 consensus_queue.cc:1035] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:55.877780 14660 consensus_queue.cc:1035] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:55.908893 6734 connection.cc:535] client connection to 127.0.186.6:33971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.909143 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.909986 9733 connection.cc:535] client connection to 127.0.186.6:33971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.910269 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.913509 6865 connection.cc:535] client connection to 127.0.186.6:33971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.913769 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.914237 6645 connection.cc:535] server connection from 127.0.186.6:42117 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.914767 6995 connection.cc:535] client connection to 127.0.186.6:33971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.915009 6995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.915087 7125 connection.cc:535] client connection to 127.0.186.6:33971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.915336 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.916179 14475 tablet_copy_client.cc:1108] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c: tablet copy: Unable to close tablet copy session: Network error: failure ending tablet copy session: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.919335 9732 connection.cc:535] server connection from 127.0.186.6:51981 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.923420 9863 connection.cc:535] client connection to 127.0.186.6:33971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:55.923729 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:55.924465 6798 raft_consensus.cc:2932] T f47877cdf5f04676861b3dc4708f3920 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:55.961174 6655 catalog_manager.cc:5484] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:13:55.962258 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:55.962457 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:55.962584 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:55.962678 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:55.966352 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f47877cdf5f04676861b3dc4708f3920 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 14:13:55.966784 6734 consensus_peers.cc:485] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:55.966910 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:55.970607 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet f47877cdf5f04676861b3dc4708f3920 with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:55.976784 14764 tablet_replica.cc:330] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:55.977046 14764 raft_consensus.cc:2229] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:55.977329 14764 raft_consensus.cc:2258] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:55.984817 14653 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap starting.
W20240503 14:13:55.990329 9467 connection.cc:535] client connection to 127.0.186.6:33971 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:55.993580 14130 consensus_queue.cc:1035] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
W20240503 14:13:55.995208 9467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 14:13:55.996646 14670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ecf4e868774482b4600566943fd224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:55.996165 9532 raft_consensus.cc:2932] T 5dc7cefd974149148dcf856a50ea168c P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:56.000789 14764 ts_tablet_manager.cc:1875] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.005990 9529 raft_consensus.cc:1261] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:56.011152 14689 consensus_queue.cc:1035] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:56.023455 14653 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.023891 14653 tablet_bootstrap.cc:492] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: Bootstrap complete.
I20240503 14:13:56.024178 14653 ts_tablet_manager.cc:1367] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.002s
I20240503 14:13:56.028052 14488 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.028507 14488 tablet_bootstrap.cc:492] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: Bootstrap complete.
I20240503 14:13:56.028828 14488 ts_tablet_manager.cc:1367] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.009s
I20240503 14:13:56.029650 14488 raft_consensus.cc:348] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:56.029923 14488 raft_consensus.cc:729] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Initialized, Role: LEARNER
W20240503 14:13:56.029997 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.030134 14488 consensus_queue.cc:260] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } }
I20240503 14:13:56.034591 6798 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:56.024947 14653 raft_consensus.cc:348] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:56.038684 14653 raft_consensus.cc:729] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Initialized, Role: LEARNER
I20240503 14:13:56.038933 14653 consensus_queue.cc:260] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: NON_VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: true } }
I20240503 14:13:56.039564 14653 ts_tablet_manager.cc:1398] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
W20240503 14:13:56.031315 9733 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 121098us
I20240503 14:13:56.044867 14488 ts_tablet_manager.cc:1398] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 14:13:56.045692 7211 tablet_copy_service.cc:342] P e1bf5fb7402c46829974e2f74cd88708: Request end of tablet copy session f1a669e28f014fe287932acf56bebc13-1708b19d3c5c4d308f4358bd42182c80 received from {username='slave'} at 127.0.186.1:54537
I20240503 14:13:56.045820 7211 tablet_copy_service.cc:433] P e1bf5fb7402c46829974e2f74cd88708: ending tablet copy session f1a669e28f014fe287932acf56bebc13-1708b19d3c5c4d308f4358bd42182c80 on tablet 1708b19d3c5c4d308f4358bd42182c80 with peer f1a669e28f014fe287932acf56bebc13
I20240503 14:13:56.050596 6929 raft_consensus.cc:1261] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:56.029458 14725 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
W20240503 14:13:56.070466 9469 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.3:44691, user_credentials={real_user=slave}} blocked reactor thread for 83599.1us
I20240503 14:13:56.076547 6651 catalog_manager.cc:5484] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:56.082975 7190 consensus_queue.cc:237] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:56.089030 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session e1bf5fb7402c46829974e2f74cd88708-92b477db785d47749fabf31ef385e9d2 received from {username='slave'} at 127.0.186.4:48587
I20240503 14:13:56.089212 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session e1bf5fb7402c46829974e2f74cd88708-92b477db785d47749fabf31ef385e9d2 on tablet 92b477db785d47749fabf31ef385e9d2 with peer e1bf5fb7402c46829974e2f74cd88708
I20240503 14:13:56.092649 6798 raft_consensus.cc:1261] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.095580 14738 ts_tablet_manager.cc:1888] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:56.095814 14738 log.cc:1196] T 61b62d3aa0bc46c3be67eadc21466ec6 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/61b62d3aa0bc46c3be67eadc21466ec6
I20240503 14:13:56.096968 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 61b62d3aa0bc46c3be67eadc21466ec6 (table rebalancer_extra_table_44 [id=194c7e8fab284faf9d2f0b8dd3d0c276]) successfully deleted
I20240503 14:13:56.104295 9529 raft_consensus.cc:1261] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.107296 9532 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:56.111783 14723 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:56.114081 14723 consensus_queue.cc:1035] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.117681 7190 raft_consensus.cc:1203] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:56.130724 14723 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:56.131141 6798 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:56.154503 14670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.186.1:33205 (request call id 197) took 1101 ms. Trace:
I20240503 14:13:56.154810 14670 rpcz_store.cc:276] 0503 14:13:55.055738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0503 14:13:55.056290 (+ 552us) service_pool.cc:225] Handling call
0503 14:13:55.085025 (+ 28735us) ts_tablet_manager.cc:903] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: Initiating tablet copy from peer f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
0503 14:13:55.898282 (+813257us) tablet_copy_client.cc:353] Tablet copy session begun
0503 14:13:56.052112 (+153830us) tablet_metadata.cc:641] Metadata flushed
0503 14:13:56.052237 (+ 125us) tablet_copy_client.cc:482] Wrote new tablet metadata
0503 14:13:56.154108 (+101871us) tablet_copy_client.cc:403] Wrote new consensus metadata
0503 14:13:56.154266 (+ 158us) tablet_replica.cc:205] Creating consensus instance
0503 14:13:56.154416 (+ 150us) ts_tablet_manager.cc:958] Replica registered.
0503 14:13:56.154445 (+ 29us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":118,"tablet-copy.queue_time_us":28329,"thread_start_us":182052,"threads_started":5}
I20240503 14:13:56.158494 6651 catalog_manager.cc:5484] T 16b01e1bab7949cfb014c446c08434a8 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:56.163652 9532 raft_consensus.cc:2932] T 16b01e1bab7949cfb014c446c08434a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 3efdbd3506274dba846ae40404f0572c (127.0.186.7) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:56.178633 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 16b01e1bab7949cfb014c446c08434a8 with delete_type TABLET_DATA_TOMBSTONED (TS 3efdbd3506274dba846ae40404f0572c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:51640
I20240503 14:13:56.183897 14724 raft_consensus.cc:482] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.184150 14724 raft_consensus.cc:504] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:56.185288 14724 leader_election.cc:288] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:56.189968 14764 ts_tablet_manager.cc:1888] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:56.190214 14764 log.cc:1196] T f47877cdf5f04676861b3dc4708f3920 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/f47877cdf5f04676861b3dc4708f3920
I20240503 14:13:56.191442 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet f47877cdf5f04676861b3dc4708f3920 (table rebalancer_extra_table_50 [id=25b71929c298472780763eadac81fe4d]) successfully deleted
I20240503 14:13:56.192109 14764 tablet_replica.cc:330] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: stopping tablet replica
I20240503 14:13:56.192454 14764 raft_consensus.cc:2229] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:56.192719 14764 pending_rounds.cc:62] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: Trying to abort 1 pending ops.
I20240503 14:13:56.193315 14764 pending_rounds.cc:68] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023598123419246592 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "16b01e1bab7949cfb014c446c08434a8" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: NON_VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } } }
I20240503 14:13:56.194006 14764 raft_consensus.cc:2894] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 14:13:56.198572 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16b01e1bab7949cfb014c446c08434a8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:56.212921 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565296ab89c24009835aefefa06fd205" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:56.221594 14670 tablet_copy_client.cc:783] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:56.221984 14670 tablet_copy_client.cc:647] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Starting download of 1 WAL segments...
W20240503 14:13:56.230633 7125 leader_election.cc:334] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:56.230901 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:56.230978 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.231066 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.231156 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.231245 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.231334 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.231423 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.232975 14745 raft_consensus.cc:2932] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
W20240503 14:13:56.234874 6865 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 321158us
I20240503 14:13:56.236501 7190 raft_consensus.cc:2932] T e14af3987fbc44e896615d5229d815d7 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:56.233834 14297 consensus_queue.cc:1035] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:56.240037 14764 raft_consensus.cc:2866] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 14:13:56.240208 14764 raft_consensus.cc:2258] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:56.234725 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.241750 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.241902 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.242934 14764 ts_tablet_manager.cc:1875] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 14:13:56.247702 9467 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.6:33971, user_credentials={real_user=slave}} blocked reactor thread for 254559us
W20240503 14:13:56.250109 6995 consensus_peers.cc:485] T 1d3ec4f0833d4d489f774757312efb32 P 09723f236d8044538d920a37b5d9aa8c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.250353 6995 consensus_peers.cc:485] T a44ed92d65114bfda2e0f38054b17ada P 09723f236d8044538d920a37b5d9aa8c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.250788 9733 consensus_peers.cc:485] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.251178 14045 raft_consensus.cc:482] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
W20240503 14:13:56.251420 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.251511 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.251349 14045 raft_consensus.cc:504] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:56.252166 14045 leader_election.cc:288] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:13:56.254330 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.254455 6865 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.254525 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.254585 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.254643 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.254702 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.254760 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.256057 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:56.263021 7061 raft_consensus.cc:2932] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:56.274197 14670 tablet_copy_client.cc:523] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 14:13:56.278993 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.279493 14681 raft_consensus.cc:482] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.279685 14681 raft_consensus.cc:504] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.280373 14681 leader_election.cc:288] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.281316 14764 ts_tablet_manager.cc:1888] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:56.281536 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461e0716aee4f74acefc639cf79eaf0" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
W20240503 14:13:56.282361 6995 leader_election.cc:334] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.281088 14670 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap starting.
W20240503 14:13:56.288340 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.288981 6995 leader_election.cc:302] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c; no voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.289520 14681 raft_consensus.cc:2726] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:13:56.289547 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.289685 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:56.291461 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.306937 14764 log.cc:1196] T 16b01e1bab7949cfb014c446c08434a8 P 3efdbd3506274dba846ae40404f0572c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-6/wal/wals/16b01e1bab7949cfb014c446c08434a8
I20240503 14:13:56.307029 6651 catalog_manager.cc:5484] T e14af3987fbc44e896615d5229d815d7 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: config changed from index 4 to 5, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:56.308843 7127 leader_election.cc:302] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5
W20240503 14:13:56.290408 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.315855 6645 catalog_manager.cc:4830] TS 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783): tablet 16b01e1bab7949cfb014c446c08434a8 (table rebalancer_extra_table_48 [id=2edf7122fe5f49728a8ab01ae30d8ea8]) successfully deleted
W20240503 14:13:56.316710 9733 leader_election.cc:334] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.317318 14723 raft_consensus.cc:2726] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:56.318235 9733 leader_election.cc:302] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: e1bf5fb7402c46829974e2f74cd88708, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.319921 14759 raft_consensus.cc:2726] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:13:56.342237 14779 raft_consensus.cc:2932] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:56.344545 14670 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:56.345124 14670 tablet_bootstrap.cc:492] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: Bootstrap complete.
I20240503 14:13:56.345443 14670 ts_tablet_manager.cc:1367] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: Time spent bootstrapping tablet: real 0.066s user 0.019s sys 0.000s
I20240503 14:13:56.346438 14670 raft_consensus.cc:348] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:56.346870 14670 raft_consensus.cc:729] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Initialized, Role: LEARNER
I20240503 14:13:56.347172 14670 consensus_queue.cc:260] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: NON_VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: true } }
I20240503 14:13:56.347307 9528 raft_consensus.cc:2932] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:56.348076 14670 ts_tablet_manager.cc:1398] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 14:13:56.347283 6929 raft_consensus.cc:2932] T 9e5922e2c6624846b64abf6df41a5601 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:56.349990 14723 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.350154 14723 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.350863 14723 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:56.351555 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458ebc1295ee49999586eb699a29bbf8" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:56.351814 6820 tablet_copy_service.cc:342] P f1a669e28f014fe287932acf56bebc13: Request end of tablet copy session 5b7e36bf354f480c90859cb8867b59cf-00ecf4e868774482b4600566943fd224 received from {username='slave'} at 127.0.186.5:35165
I20240503 14:13:56.351925 6820 tablet_copy_service.cc:433] P f1a669e28f014fe287932acf56bebc13: ending tablet copy session 5b7e36bf354f480c90859cb8867b59cf-00ecf4e868774482b4600566943fd224 on tablet 00ecf4e868774482b4600566943fd224 with peer 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:56.360761 14620 raft_consensus.cc:482] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.360996 14620 raft_consensus.cc:504] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:56.361763 14620 leader_election.cc:288] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.361793 6654 catalog_manager.cc:5484] T 9e5922e2c6624846b64abf6df41a5601 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:56.362608 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:56.362825 9528 raft_consensus.cc:2454] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:56.364473 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
W20240503 14:13:56.372182 6734 leader_election.cc:334] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.372332 6734 leader_election.cc:302] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters: ea801601713a4d55aee0de9e3473e0bc, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.372887 14620 raft_consensus.cc:2726] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:13:56.374082 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.374191 7125 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.375993 14725 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:56.380273 14045 raft_consensus.cc:482] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.380456 14045 raft_consensus.cc:504] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:56.381080 14045 leader_election.cc:288] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.382469 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:56.383837 6797 consensus_queue.cc:237] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
W20240503 14:13:56.384047 9733 leader_election.cc:334] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.384207 9733 leader_election.cc:302] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.384629 14045 raft_consensus.cc:2726] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:13:56.386329 9528 raft_consensus.cc:1261] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.387219 14620 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.390272 6930 raft_consensus.cc:1261] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.390290 14624 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:56.391067 14378 consensus_queue.cc:1035] T d1df9d5b8a3d4a489cbccca67b223168 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.390969 9528 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:56.392490 14446 raft_consensus.cc:482] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.392659 14446 raft_consensus.cc:504] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:56.393467 14446 leader_election.cc:288] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:56.394011 6797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:56.394081 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:56.394280 9532 raft_consensus.cc:2454] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:56.394271 6797 raft_consensus.cc:2454] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
I20240503 14:13:56.394783 9863 leader_election.cc:302] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:56.395265 14446 raft_consensus.cc:2781] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.395385 14446 raft_consensus.cc:482] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.395509 14446 raft_consensus.cc:3037] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.396039 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d1df9d5b8a3d4a489cbccca67b223168 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:56.397790 14446 raft_consensus.cc:504] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
W20240503 14:13:56.398154 9863 leader_election.cc:334] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.399919 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:56.400111 9532 raft_consensus.cc:3037] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.400359 14446 leader_election.cc:288] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
W20240503 14:13:56.402174 9863 leader_election.cc:334] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.402779 9532 raft_consensus.cc:2454] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:56.404605 6653 catalog_manager.cc:5484] T d1df9d5b8a3d4a489cbccca67b223168 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:56.406868 6797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3f2bd341493482f9519e6dd74cd2f3d" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:56.407092 6797 raft_consensus.cc:3037] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.407850 6930 raft_consensus.cc:2932] T d1df9d5b8a3d4a489cbccca67b223168 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ea801601713a4d55aee0de9e3473e0bc (127.0.186.8) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:56.409879 6797 raft_consensus.cc:2454] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:56.411187 14660 raft_consensus.cc:482] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.411402 14660 raft_consensus.cc:504] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:56.412142 14660 leader_election.cc:288] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.412969 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133b4518f1f4decba0ba32cd82e528e" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:56.415864 9909 tablet_service.cc:1514] Processing DeleteTablet for tablet d1df9d5b8a3d4a489cbccca67b223168 with delete_type TABLET_DATA_TOMBSTONED (TS ea801601713a4d55aee0de9e3473e0bc not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54300
I20240503 14:13:56.417239 14791 tablet_replica.cc:330] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: stopping tablet replica
I20240503 14:13:56.417544 14791 raft_consensus.cc:2229] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:56.417863 14791 raft_consensus.cc:2258] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Raft consensus is shut down!
W20240503 14:13:56.418458 9467 leader_election.cc:334] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.418627 9467 leader_election.cc:302] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 3efdbd3506274dba846ae40404f0572c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.418771 9863 leader_election.cc:302] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.418977 14660 raft_consensus.cc:2726] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:56.422451 14446 raft_consensus.cc:2781] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.422603 14446 raft_consensus.cc:686] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:56.422925 14446 consensus_queue.cc:237] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:56.424996 14791 ts_tablet_manager.cc:1875] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.428974 6654 catalog_manager.cc:5484] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:56.433312 14791 ts_tablet_manager.cc:1888] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:56.433571 14791 log.cc:1196] T d1df9d5b8a3d4a489cbccca67b223168 P ea801601713a4d55aee0de9e3473e0bc: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-7/wal/wals/d1df9d5b8a3d4a489cbccca67b223168
I20240503 14:13:56.434602 6644 catalog_manager.cc:4830] TS ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653): tablet d1df9d5b8a3d4a489cbccca67b223168 (table rebalancer_extra_table_32 [id=dfa34e8282dc4923809ea00e1bbf73f2]) successfully deleted
I20240503 14:13:56.437024 14660 raft_consensus.cc:482] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.437273 14660 raft_consensus.cc:504] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:56.438014 14660 leader_election.cc:288] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:56.438666 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565296ab89c24009835aefefa06fd205" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:56.438877 7187 raft_consensus.cc:2454] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:56.439285 9468 leader_election.cc:302] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:56.439582 14660 raft_consensus.cc:2781] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.439713 14660 raft_consensus.cc:482] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.439837 14660 raft_consensus.cc:3037] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:56.440497 9467 leader_election.cc:334] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.441949 14660 raft_consensus.cc:504] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:56.442456 14660 leader_election.cc:288] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:56.442970 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565296ab89c24009835aefefa06fd205" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:56.443192 7187 raft_consensus.cc:3037] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:56.443850 9467 leader_election.cc:334] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.445436 7187 raft_consensus.cc:2454] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:56.446014 9468 leader_election.cc:302] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.446383 14660 raft_consensus.cc:2781] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.446540 14660 raft_consensus.cc:686] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:56.447209 14660 consensus_queue.cc:237] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } }
I20240503 14:13:56.450583 6797 raft_consensus.cc:1203] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:56.452661 6654 catalog_manager.cc:5484] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:56.481472 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.488021 14446 raft_consensus.cc:482] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.488298 14446 raft_consensus.cc:504] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.489210 14446 leader_election.cc:288] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:56.489984 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461e0716aee4f74acefc639cf79eaf0" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
I20240503 14:13:56.490272 7061 raft_consensus.cc:2454] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 1.
W20240503 14:13:56.492939 9863 leader_election.cc:334] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.494099 9865 leader_election.cc:302] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, ea801601713a4d55aee0de9e3473e0bc; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.494668 14354 raft_consensus.cc:2781] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.494781 14354 raft_consensus.cc:482] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.494905 14354 raft_consensus.cc:3037] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.497709 14354 raft_consensus.cc:504] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.498319 14354 leader_election.cc:288] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:56.498765 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1461e0716aee4f74acefc639cf79eaf0" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:56.498994 7061 raft_consensus.cc:3037] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.503742 7061 raft_consensus.cc:2454] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea801601713a4d55aee0de9e3473e0bc in term 2.
I20240503 14:13:56.504184 9865 leader_election.cc:302] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, ea801601713a4d55aee0de9e3473e0bc; no voters:
W20240503 14:13:56.505157 9863 leader_election.cc:334] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.509338 14446 raft_consensus.cc:2781] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.509496 14446 raft_consensus.cc:686] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Becoming Leader. State: Replica: ea801601713a4d55aee0de9e3473e0bc, State: Running, Role: LEADER
I20240503 14:13:56.509820 14446 consensus_queue.cc:237] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.515563 6654 catalog_manager.cc:5484] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to ea801601713a4d55aee0de9e3473e0bc (127.0.186.8). New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:56.519080 14354 raft_consensus.cc:482] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.519349 14354 raft_consensus.cc:504] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:56.520895 14354 leader_election.cc:288] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:56.522080 6797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a89b1b988df4ad8aa8f1311cfa35731" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
W20240503 14:13:56.524441 9863 leader_election.cc:334] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.524586 9863 leader_election.cc:302] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: f03dc37fe3234033bc665bed4f1b51f5, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:56.524886 14354 raft_consensus.cc:2726] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:56.529758 6929 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9"
dest_uuid: "7186ed8510424fd597173eaf10c57b12"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46796
I20240503 14:13:56.529953 6929 raft_consensus.cc:595] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Received request to transfer leadership
I20240503 14:13:56.534835 14725 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.535053 14725 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.535784 14725 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:56.536525 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "09723f236d8044538d920a37b5d9aa8c" is_pre_election: true
W20240503 14:13:56.537321 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.537454 7125 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.537686 14725 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:56.544476 14576 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.544667 14576 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.545332 14576 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:56.546023 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458ebc1295ee49999586eb699a29bbf8" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:56.546290 7187 raft_consensus.cc:2454] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
W20240503 14:13:56.546918 6995 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.547098 6996 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.547609 14576 raft_consensus.cc:2781] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.547726 14576 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.547814 14576 raft_consensus.cc:3037] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.550559 14576 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.551640 14576 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:56.552443 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "458ebc1295ee49999586eb699a29bbf8" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:56.552662 7187 raft_consensus.cc:3037] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:56.556185 6995 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.559399 14354 raft_consensus.cc:482] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.559679 14354 raft_consensus.cc:504] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:56.560297 14389 raft_consensus.cc:979] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12: : Instructing follower f1a669e28f014fe287932acf56bebc13 to start an election
I20240503 14:13:56.560418 14389 raft_consensus.cc:1067] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Signalling peer f1a669e28f014fe287932acf56bebc13 to start an election
I20240503 14:13:56.560472 14354 leader_election.cc:288] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.561364 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f54c0c48b848ed898534efef55fb62" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
I20240503 14:13:56.562393 6797 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9"
dest_uuid: "f1a669e28f014fe287932acf56bebc13"
from {username='slave'} at 127.0.186.2:33179
I20240503 14:13:56.562611 6797 raft_consensus.cc:482] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 14:13:56.562727 6797 raft_consensus.cc:3037] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.563949 7187 raft_consensus.cc:2454] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 2.
I20240503 14:13:56.564529 6996 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.564888 14681 raft_consensus.cc:2781] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election won for term 2
W20240503 14:13:56.565604 9863 leader_election.cc:334] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.565690 14681 raft_consensus.cc:686] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:56.565755 9863 leader_election.cc:302] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: 7186ed8510424fd597173eaf10c57b12, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.566080 14354 raft_consensus.cc:2726] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:56.566090 14681 consensus_queue.cc:237] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.568331 6797 raft_consensus.cc:504] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:56.569612 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "7186ed8510424fd597173eaf10c57b12"
I20240503 14:13:56.569820 6929 raft_consensus.cc:3032] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 14:13:56.569908 6929 raft_consensus.cc:729] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:56.570183 6929 consensus_queue.cc:260] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:56.570598 6929 raft_consensus.cc:3037] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.572058 6654 catalog_manager.cc:5484] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 2 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:56.573323 6929 raft_consensus.cc:2454] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:56.576133 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "933c08460b9745feab9bcb56c27bdbb9" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "09723f236d8044538d920a37b5d9aa8c"
I20240503 14:13:56.576388 7061 raft_consensus.cc:3037] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.579205 7061 raft_consensus.cc:2454] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:56.579715 6736 leader_election.cc:302] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:56.580179 6797 leader_election.cc:288] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.583954 14624 raft_consensus.cc:2781] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.584103 14624 raft_consensus.cc:686] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [term 2 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:56.584358 14624 consensus_queue.cc:237] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
W20240503 14:13:56.590684 6734 leader_election.cc:334] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.591420 6654 catalog_manager.cc:5484] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 1 to 2, leader changed from 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 2 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:56.593881 9532 raft_consensus.cc:1203] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 14:13:56.608517 14725 raft_consensus.cc:482] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.608747 14725 raft_consensus.cc:504] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:56.609753 14725 leader_election.cc:288] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:56.610555 14130 raft_consensus.cc:482] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.610745 14130 raft_consensus.cc:504] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
W20240503 14:13:56.611097 7125 leader_election.cc:334] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.611505 14130 leader_election.cc:288] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.612315 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:56.612543 9796 raft_consensus.cc:2454] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:56.612954 9468 leader_election.cc:302] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:56.613276 14130 raft_consensus.cc:2781] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:13:56.613420 14130 raft_consensus.cc:482] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.613520 14130 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:56.613655 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:56.613876 9796 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 2.
I20240503 14:13:56.614260 7126 leader_election.cc:302] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters: f03dc37fe3234033bc665bed4f1b51f5
W20240503 14:13:56.614353 9467 leader_election.cc:334] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.614972 14724 raft_consensus.cc:2781] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 14:13:56.615044 14724 raft_consensus.cc:482] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.615039 14389 raft_consensus.cc:482] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.615103 14724 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:56.615191 14389 raft_consensus.cc:504] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:56.616003 14389 leader_election.cc:288] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:56.616003 14723 raft_consensus.cc:482] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.616151 14723 raft_consensus.cc:504] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:56.616698 14723 leader_election.cc:288] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:56.617698 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f54c0c48b848ed898534efef55fb62" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:56.617745 14724 raft_consensus.cc:504] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:56.617964 9928 raft_consensus.cc:2454] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:56.618654 6865 leader_election.cc:302] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
I20240503 14:13:56.619685 14389 raft_consensus.cc:2781] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.619767 14389 raft_consensus.cc:482] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.619834 14389 raft_consensus.cc:3037] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:56.619804 6865 leader_election.cc:334] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:56.620242 7125 leader_election.cc:334] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.621587 14130 raft_consensus.cc:504] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:56.622682 9796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77d3f36de08d4f1986e42e86e1b95fbf" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:56.622908 9796 raft_consensus.cc:3037] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:56.623348 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:56.623924 14389 raft_consensus.cc:504] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:56.624641 14389 leader_election.cc:288] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:56.624743 14724 leader_election.cc:288] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:56.624861 7127 leader_election.cc:302] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 5b7e36bf354f480c90859cb8867b59cf, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.624859 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b154f780fc742fa9b0efd609c307f95" candidate_uuid: "e1bf5fb7402c46829974e2f74cd88708" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:56.625010 9799 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Advancing to term 3
I20240503 14:13:56.625098 14724 raft_consensus.cc:2726] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:56.625906 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2f54c0c48b848ed898534efef55fb62" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:56.626116 9928 raft_consensus.cc:3037] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.627128 9799 raft_consensus.cc:2454] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1bf5fb7402c46829974e2f74cd88708 in term 3.
I20240503 14:13:56.627518 7126 leader_election.cc:302] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:56.627903 14724 raft_consensus.cc:2781] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:13:56.628078 14724 raft_consensus.cc:686] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 3 LEADER]: Becoming Leader. State: Replica: e1bf5fb7402c46829974e2f74cd88708, State: Running, Role: LEADER
I20240503 14:13:56.628439 14724 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
W20240503 14:13:56.628933 7125 leader_election.cc:334] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.629164 9928 raft_consensus.cc:2454] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 2.
I20240503 14:13:56.629590 6865 leader_election.cc:302] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters:
W20240503 14:13:56.629815 6865 leader_election.cc:334] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.630133 9796 raft_consensus.cc:2454] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 3.
I20240503 14:13:56.630542 9468 leader_election.cc:302] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:56.631114 14660 raft_consensus.cc:2781] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 3 FOLLOWER]: Leader election won for term 3
I20240503 14:13:56.631268 14660 raft_consensus.cc:686] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [term 3 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:56.631558 14660 consensus_queue.cc:237] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:56.631747 14389 raft_consensus.cc:2781] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.631951 14389 raft_consensus.cc:686] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 2 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:56.632217 14389 consensus_queue.cc:237] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } }
I20240503 14:13:56.637537 14446 raft_consensus.cc:1050] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER e1bf5fb7402c46829974e2f74cd88708 to VOTER
W20240503 14:13:56.637727 9467 leader_election.cc:334] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.638446 14446 consensus_queue.cc:237] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:56.641537 6653 catalog_manager.cc:5484] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 2 to 3, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 3 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:56.644876 14130 leader_election.cc:288] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.647572 14576 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.647823 14576 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.648654 14576 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:56.649510 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:56.649710 7187 raft_consensus.cc:2454] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 1.
I20240503 14:13:56.650144 6996 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:56.650465 14576 raft_consensus.cc:2781] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.650564 14576 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.650657 14576 raft_consensus.cc:3037] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:56.650770 6995 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.653666 14576 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.654640 7187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62d262e208a84c14bd296e0c44947813" candidate_uuid: "09723f236d8044538d920a37b5d9aa8c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:56.654834 7187 raft_consensus.cc:3037] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.654937 14130 raft_consensus.cc:482] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.655104 14130 raft_consensus.cc:504] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:56.655793 14130 leader_election.cc:288] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:56.657044 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708" is_pre_election: true
I20240503 14:13:56.657253 7190 raft_consensus.cc:2454] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:56.657652 7187 raft_consensus.cc:2454] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09723f236d8044538d920a37b5d9aa8c in term 2.
I20240503 14:13:56.657269 6653 catalog_manager.cc:5484] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 2 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:56.658073 6996 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09723f236d8044538d920a37b5d9aa8c, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:56.658160 9468 leader_election.cc:302] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters:
I20240503 14:13:56.658388 14834 raft_consensus.cc:2781] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.658561 14834 raft_consensus.cc:686] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEADER]: Becoming Leader. State: Replica: 09723f236d8044538d920a37b5d9aa8c, State: Running, Role: LEADER
I20240503 14:13:56.658879 14834 consensus_queue.cc:237] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:56.660331 14576 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
W20240503 14:13:56.665151 9467 leader_election.cc:334] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.665699 14130 raft_consensus.cc:2781] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.665812 14130 raft_consensus.cc:482] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.665897 14130 raft_consensus.cc:3037] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:56.669814 6995 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.671624 14130 raft_consensus.cc:504] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:56.671911 7187 raft_consensus.cc:1261] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:56.672995 14354 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240503 14:13:56.674419 6797 raft_consensus.cc:1261] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:56.675436 14731 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:56.676026 7061 raft_consensus.cc:1261] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:56.677402 14731 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:56.677752 6653 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: term changed from 2 to 3, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4). New cstate: current_term: 3 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:56.678159 9467 leader_election.cc:334] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.678841 7190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "603002fdf6b84d3ea63c65636eb42cb1" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e1bf5fb7402c46829974e2f74cd88708"
I20240503 14:13:56.678895 14378 raft_consensus.cc:482] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.679047 7190 raft_consensus.cc:3037] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.679049 14378 raft_consensus.cc:504] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:56.679865 14378 leader_election.cc:288] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.682440 7190 raft_consensus.cc:2454] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:56.684203 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a89b1b988df4ad8aa8f1311cfa35731" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:56.684415 9928 raft_consensus.cc:2454] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:56.687172 9468 leader_election.cc:302] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, e1bf5fb7402c46829974e2f74cd88708; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.687492 14660 raft_consensus.cc:2781] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.687651 14660 raft_consensus.cc:686] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:56.687929 14660 consensus_queue.cc:237] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
I20240503 14:13:56.691489 14446 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:56.697252 7190 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:56.699331 7061 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:56.702878 14779 raft_consensus.cc:482] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.703080 14779 raft_consensus.cc:504] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:56.703722 14779 leader_election.cc:288] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.707805 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133b4518f1f4decba0ba32cd82e528e" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:56.708070 9528 raft_consensus.cc:2454] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 1.
I20240503 14:13:56.708961 6655 catalog_manager.cc:5484] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:56.711036 9735 leader_election.cc:302] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:56.711314 14045 raft_consensus.cc:2781] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.711436 14045 raft_consensus.cc:482] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.711519 14045 raft_consensus.cc:3037] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:56.713233 9733 leader_election.cc:334] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.718827 14130 leader_election.cc:288] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Requested vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
I20240503 14:13:56.686004 6734 leader_election.cc:302] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters:
W20240503 14:13:56.740573 6734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 54584.9us
I20240503 14:13:56.742097 14624 raft_consensus.cc:2781] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:56.742218 14624 raft_consensus.cc:482] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f03dc37fe3234033bc665bed4f1b51f5)
I20240503 14:13:56.742281 14624 raft_consensus.cc:3037] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.750362 6654 catalog_manager.cc:5484] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to 09723f236d8044538d920a37b5d9aa8c (127.0.186.3). New cstate: current_term: 2 leader_uuid: "09723f236d8044538d920a37b5d9aa8c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:56.754593 6655 catalog_manager.cc:5484] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:56.757639 14624 raft_consensus.cc:504] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:56.758524 14624 leader_election.cc:288] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Requested vote from peers ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.774724 14045 raft_consensus.cc:504] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:56.777568 14045 leader_election.cc:288] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 election: Requested vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:56.780486 9528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d133b4518f1f4decba0ba32cd82e528e" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:56.780723 9528 raft_consensus.cc:3037] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.781862 6797 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:56.782899 9528 raft_consensus.cc:2454] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3efdbd3506274dba846ae40404f0572c in term 2.
W20240503 14:13:56.783205 9733 leader_election.cc:334] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.783394 9735 leader_election.cc:302] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters: f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:56.783844 14779 raft_consensus.cc:2781] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.783993 14779 raft_consensus.cc:686] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 2 LEADER]: Becoming Leader. State: Replica: 3efdbd3506274dba846ae40404f0572c, State: Running, Role: LEADER
I20240503 14:13:56.784256 14779 consensus_queue.cc:237] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } }
I20240503 14:13:56.810319 9928 consensus_queue.cc:237] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } }
W20240503 14:13:56.812470 6734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.186.8:36653, user_credentials={real_user=slave}} blocked reactor thread for 70880us
W20240503 14:13:56.813824 6734 leader_election.cc:334] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:56.815032 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a89b1b988df4ad8aa8f1311cfa35731" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:56.815392 9928 raft_consensus.cc:3037] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:56.815840 7190 raft_consensus.cc:1261] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.817277 14731 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.820221 14731 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:56.821099 9928 raft_consensus.cc:2454] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:56.826225 7190 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:56.826957 6734 leader_election.cc:302] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:56.827250 14378 raft_consensus.cc:2781] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:56.827387 14378 raft_consensus.cc:686] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [term 2 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:56.827816 14378 consensus_queue.cc:237] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } }
I20240503 14:13:56.829175 6797 raft_consensus.cc:1261] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:56.830921 14446 consensus_queue.cc:1035] T 92b477db785d47749fabf31ef385e9d2 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.831982 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 92b477db785d47749fabf31ef385e9d2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:56.832566 6797 raft_consensus.cc:2932] T 92b477db785d47749fabf31ef385e9d2 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:56.833984 6654 catalog_manager.cc:5484] T 92b477db785d47749fabf31ef385e9d2 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 3 to 4, VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } }
I20240503 14:13:56.844167 7039 tablet_service.cc:1514] Processing DeleteTablet for tablet 92b477db785d47749fabf31ef385e9d2 with delete_type TABLET_DATA_TOMBSTONED (TS 09723f236d8044538d920a37b5d9aa8c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36700
I20240503 14:13:56.856679 6654 catalog_manager.cc:5484] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to 3efdbd3506274dba846ae40404f0572c (127.0.186.7). New cstate: current_term: 2 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:56.868206 6800 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:56.869406 14446 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.877774 6654 catalog_manager.cc:5484] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 1 to 2, leader changed from f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 2 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } }
W20240503 14:13:56.856766 6734 leader_election.cc:334] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:56.919507 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.928117 14565 raft_consensus.cc:1050] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER f1a669e28f014fe287932acf56bebc13 to VOTER
I20240503 14:13:56.929085 14565 consensus_queue.cc:237] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:56.929919 14864 tablet_replica.cc:330] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: stopping tablet replica
I20240503 14:13:56.933782 14864 raft_consensus.cc:2229] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:56.934221 14864 raft_consensus.cc:2258] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:56.935341 7061 raft_consensus.cc:1261] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:56.935482 6800 raft_consensus.cc:1261] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 [term 1 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:56.935961 14723 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:56.936280 14725 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:56.936374 14864 ts_tablet_manager.cc:1875] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:56.937384 6929 raft_consensus.cc:1261] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:56.938544 14725 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:56.939096 7056 raft_consensus.cc:1261] T 1461e0716aee4f74acefc639cf79eaf0 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:56.940327 14731 consensus_queue.cc:1035] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:56.954560 9863 consensus_peers.cc:485] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.956923 9528 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:56.957706 14446 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:56.966070 14723 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:56.971520 6930 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:56.975159 7190 raft_consensus.cc:1261] T 565296ab89c24009835aefefa06fd205 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:13:56.976143 14660 consensus_queue.cc:1035] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:56.977922 6797 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
W20240503 14:13:56.982901 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:56.983364 6655 catalog_manager.cc:5484] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.010092 7061 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f1a669e28f014fe287932acf56bebc13 (127.0.186.1) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.010465 6929 raft_consensus.cc:1261] T 933c08460b9745feab9bcb56c27bdbb9 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:57.011054 14378 consensus_queue.cc:1035] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.021098 7187 consensus_queue.cc:237] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:57.023411 7056 raft_consensus.cc:1261] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:57.024029 6797 raft_consensus.cc:1261] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:57.024721 14724 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.025513 14724 consensus_queue.cc:1035] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.027467 14860 raft_consensus.cc:1050] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13: attempting to promote NON_VOTER 5b7e36bf354f480c90859cb8867b59cf to VOTER
I20240503 14:13:57.028319 14860 consensus_queue.cc:237] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:57.028163 14723 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.029305 6797 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.041595 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1708b19d3c5c4d308f4358bd42182c80 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 14:13:57.046233 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.046989 7057 raft_consensus.cc:2932] T 1708b19d3c5c4d308f4358bd42182c80 P 09723f236d8044538d920a37b5d9aa8c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.056051 7191 raft_consensus.cc:1261] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:57.057133 14834 consensus_queue.cc:1035] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:57.059231 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:57.063369 6995 consensus_peers.cc:485] T 458ebc1295ee49999586eb699a29bbf8 P 09723f236d8044538d920a37b5d9aa8c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:57.069789 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.086478 14864 ts_tablet_manager.cc:1888] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:57.086629 14864 log.cc:1196] T 92b477db785d47749fabf31ef385e9d2 P 09723f236d8044538d920a37b5d9aa8c: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-2/wal/wals/92b477db785d47749fabf31ef385e9d2
I20240503 14:13:57.087283 6646 catalog_manager.cc:4830] TS 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): tablet 92b477db785d47749fabf31ef385e9d2 (table rebalancer_extra_table_46 [id=bf90396bdf234cb2adab81e6a7f8ec52]) successfully deleted
I20240503 14:13:57.088438 6654 catalog_manager.cc:5484] T 1708b19d3c5c4d308f4358bd42182c80 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 3 to 4, VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) evicted. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.095801 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet 1708b19d3c5c4d308f4358bd42182c80 with delete_type TABLET_DATA_TOMBSTONED (TS 7186ed8510424fd597173eaf10c57b12 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:34064
I20240503 14:13:57.096585 14883 tablet_replica.cc:330] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: stopping tablet replica
I20240503 14:13:57.096782 14883 raft_consensus.cc:2229] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:57.096985 14883 raft_consensus.cc:2258] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:57.097975 14883 ts_tablet_manager.cc:1875] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:57.101290 9796 raft_consensus.cc:1261] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:57.102123 14378 consensus_queue.cc:1035] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:13:57.102492 6734 consensus_peers.cc:485] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.111832 9928 raft_consensus.cc:1261] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:13:57.113128 7059 raft_consensus.cc:3037] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 2 LEARNER]: Advancing to term 3
I20240503 14:13:57.113049 14379 consensus_queue.cc:1035] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:57.114450 9528 raft_consensus.cc:1261] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf [term 1 LEARNER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:57.115558 14860 consensus_queue.cc:1035] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:57.116250 7059 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 3 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 14:13:57.118832 14565 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240503 14:13:57.120633 9799 raft_consensus.cc:3037] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c [term 1 LEARNER]: Advancing to term 2
W20240503 14:13:57.122848 6995 consensus_peers.cc:485] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.123374 7187 raft_consensus.cc:1261] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 09723f236d8044538d920a37b5d9aa8c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:57.124389 14681 consensus_queue.cc:1035] T 62d262e208a84c14bd296e0c44947813 P 09723f236d8044538d920a37b5d9aa8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.125924 9799 raft_consensus.cc:1261] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c [term 2 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 14:13:57.126622 14379 consensus_queue.cc:1035] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: NON_VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:57.142380 14883 ts_tablet_manager.cc:1888] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 14:13:57.144881 9795 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 3 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240503 14:13:57.145599 14883 log.cc:1196] T 1708b19d3c5c4d308f4358bd42182c80 P 7186ed8510424fd597173eaf10c57b12: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-1/wal/wals/1708b19d3c5c4d308f4358bd42182c80
I20240503 14:13:57.145584 14565 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240503 14:13:57.148231 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:57.147972 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.149569 6644 catalog_manager.cc:4830] TS 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): tablet 1708b19d3c5c4d308f4358bd42182c80 (table rebalancer_extra_table_40 [id=4b124ba97338445697f641d280e31185]) successfully deleted
I20240503 14:13:57.156803 9928 consensus_queue.cc:237] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:57.160411 6800 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:57.161475 14645 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.165848 9528 raft_consensus.cc:1261] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:57.166479 14645 consensus_queue.cc:1035] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.179885 7190 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:57.179538 14446 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P ea801601713a4d55aee0de9e3473e0bc [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.180526 14660 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.184357 9530 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.184674 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3f2bd341493482f9519e6dd74cd2f3d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.193028 6652 catalog_manager.cc:5484] T c3f2bd341493482f9519e6dd74cd2f3d P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 3 to 5, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 2 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.205653 7056 raft_consensus.cc:1261] T 933c08460b9745feab9bcb56c27bdbb9 P 09723f236d8044538d920a37b5d9aa8c [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:57.207285 14378 consensus_queue.cc:1035] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.211936 14379 raft_consensus.cc:1050] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12: attempting to promote NON_VOTER 3efdbd3506274dba846ae40404f0572c to VOTER
I20240503 14:13:57.212733 14379 consensus_queue.cc:237] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
W20240503 14:13:57.220988 6644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:57.228354 9796 raft_consensus.cc:1261] T 77d3f36de08d4f1986e42e86e1b95fbf P 3efdbd3506274dba846ae40404f0572c [term 3 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240503 14:13:57.228983 14660 consensus_queue.cc:1035] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:57.230334 9795 raft_consensus.cc:1261] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c [term 2 LEARNER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:57.230201 14378 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:57.231292 14721 consensus_queue.cc:1035] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 14:13:57.240024 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.240232 6797 raft_consensus.cc:2932] T c3f2bd341493482f9519e6dd74cd2f3d P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } } }
I20240503 14:13:57.242123 9530 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
W20240503 14:13:57.245471 9467 consensus_peers.cc:485] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.235069 9796 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
W20240503 14:13:57.254390 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3f2bd341493482f9519e6dd74cd2f3d: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:57.279937 14725 raft_consensus.cc:1050] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708: attempting to promote NON_VOTER 09723f236d8044538d920a37b5d9aa8c to VOTER
I20240503 14:13:57.280880 14725 consensus_queue.cc:237] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:57.287091 9928 raft_consensus.cc:1261] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 14:13:57.288074 9797 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 3 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
W20240503 14:13:57.288089 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.289304 14565 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:13:57.303326 14721 consensus_queue.cc:1035] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.305797 9928 raft_consensus.cc:1261] T 9a89b1b988df4ad8aa8f1311cfa35731 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:13:57.306412 14378 consensus_queue.cc:1035] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:57.311120 9532 raft_consensus.cc:1261] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 14:13:57.311903 14759 consensus_queue.cc:1035] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:57.313148 7057 raft_consensus.cc:1261] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 3 LEARNER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240503 14:13:57.313961 14565 consensus_queue.cc:1035] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240503 14:13:57.321146 7057 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 09723f236d8044538d920a37b5d9aa8c [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:57.332348 9799 raft_consensus.cc:2932] T f2f54c0c48b848ed898534efef55fb62 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:57.271553 6654 catalog_manager.cc:5484] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 2 to 3, 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:57.350039 6797 consensus_queue.cc:237] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:57.353893 9530 raft_consensus.cc:1261] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:57.354641 14624 consensus_queue.cc:1035] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.356159 9796 raft_consensus.cc:1261] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:57.356786 14624 consensus_queue.cc:1035] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:57.358304 14624 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:57.360040 9798 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:57.362885 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 00ecf4e868774482b4600566943fd224 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:57.363207 9530 raft_consensus.cc:2932] T 00ecf4e868774482b4600566943fd224 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } }
I20240503 14:13:57.369302 14389 raft_consensus.cc:2932] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:57.373739 14724 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:57.374595 6655 catalog_manager.cc:5484] T 00ecf4e868774482b4600566943fd224 P f1a669e28f014fe287932acf56bebc13 reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:13:57.381270 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:57.386008 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.387113 9796 raft_consensus.cc:2932] T 9b154f780fc742fa9b0efd609c307f95 P 3efdbd3506274dba846ae40404f0572c [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } } }
I20240503 14:13:57.396548 6652 catalog_manager.cc:5484] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 11 to 13, 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:57.404250 9928 raft_consensus.cc:2932] T f2f54c0c48b848ed898534efef55fb62 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } } }
I20240503 14:13:57.407241 6658 catalog_manager.cc:5484] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 2 to 4, 3efdbd3506274dba846ae40404f0572c (127.0.186.7) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:57.513906 7190 consensus_queue.cc:237] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:57.516515 9796 raft_consensus.cc:1261] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Refusing update from remote peer e1bf5fb7402c46829974e2f74cd88708: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:57.516870 7127 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 122296b00eda44949b9064d0afdd6a16. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.517134 14724 consensus_queue.cc:1035] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:57.517479 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.519954 14725 raft_consensus.cc:2932] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:57.520556 9796 raft_consensus.cc:2932] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
W20240503 14:13:57.525483 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:57.527046 6655 catalog_manager.cc:5484] T 122296b00eda44949b9064d0afdd6a16 P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "e1bf5fb7402c46829974e2f74cd88708" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { replace: true } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:57.548487 9928 consensus_queue.cc:237] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:57.550993 7190 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 14:13:57.551383 9863 consensus_peers.cc:485] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84b58d63957848d7a8b0eae9f60bbbf1. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.550673 9532 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 14:13:57.551739 14446 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.552289 14731 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 14:13:57.554775 14446 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:57.557228 9532 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:57.564347 7191 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:57.566020 6654 catalog_manager.cc:5484] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 4 to 5, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
W20240503 14:13:57.574424 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:57.589723 9928 consensus_queue.cc:237] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:57.592952 7191 raft_consensus.cc:1261] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:57.593272 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.593600 14446 consensus_queue.cc:1035] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 14:13:57.594107 9865 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69009d7fef744e66800ce1f07b4a62bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.596747 14645 raft_consensus.cc:2932] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:57.601881 7191 raft_consensus.cc:2932] T 69009d7fef744e66800ce1f07b4a62bf P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:57.605235 6654 catalog_manager.cc:5484] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc reported cstate change: config changed from index -1 to 2, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:57.821192 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:57.891817 9799 consensus_queue.cc:237] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } }
I20240503 14:13:57.894173 9532 raft_consensus.cc:1261] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20240503 14:13:57.894377 9735 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d133b4518f1f4decba0ba32cd82e528e. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:57.894843 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.895066 14779 consensus_queue.cc:1035] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:57.898586 14759 raft_consensus.cc:2932] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:57.899299 9532 raft_consensus.cc:2932] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:57.906791 6654 catalog_manager.cc:5484] T d133b4518f1f4decba0ba32cd82e528e P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index -1 to 3, NON_VOTER 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) added. New cstate: current_term: 2 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: NON_VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: true } } }
I20240503 14:13:57.922124 9799 consensus_queue.cc:237] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:57.924943 9929 raft_consensus.cc:1261] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 14:13:57.925155 9733 consensus_peers.cc:485] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:57.925426 9733 consensus_peers.cc:485] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbbd9f57ea9f457ab0ad4e59bc0f34bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:57.925693 14779 consensus_queue.cc:1035] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 14:13:57.928586 14759 raft_consensus.cc:2932] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:57.929185 9929 raft_consensus.cc:2932] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
W20240503 14:13:57.935927 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:57.936547 6655 catalog_manager.cc:5484] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index -1 to 2, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:57.959667 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:58.060163 15062 ts_tablet_manager.cc:902] T 69009d7fef744e66800ce1f07b4a62bf P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
F20240503 14:13:58.060499 15062 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-01e27516acca49459176362caf73595e.queue_time_us, tablet-download-pool-01e27516acca49459176362caf73595e.queue_time_us) (tablet-download-pool-01e27516acca49459176362caf73595e.run_cpu_time_us, tablet-download-pool-01e27516acca49459176362caf73595e.run_cpu_time_us) (tablet-download-pool-01e27516acca49459176362caf73595e.run_wall_time_us, tablet-download-pool-01e27516acca49459176362caf73595e.run_wall_time_us) (tablet-download-pool-10ad2f2ef32d4fc88cdd62bfeab97b8a.queue_time_us, tablet-download-pool-10ad2f2ef32d4fc88cdd62bfeab97b8a.queue_time_us) (tablet-download-pool-10ad2f2ef32d4fc88cdd62bfeab97b8a.run_cpu_time_us, tablet-download-pool-10ad2f2ef32d4fc88cdd62bfeab97b8a.run_cpu_time_us) (tablet-download-pool-10ad2f2ef32d4fc88cdd62bfeab97b8a.run_wall_time_us, tablet-download-pool-10ad2f2ef32d4fc88cdd62bfeab97b8a.run_wall_time_us) (tablet-download-pool-13f41f236987487b8bca287c7ec1fdd3.queue_time_us, tablet-download-pool-13f41f236987487b8bca287c7ec1fdd3.queue_time_us) (tablet-download-pool-13f41f236987487b8bca287c7ec1fdd3.run_cpu_time_us, tablet-download-pool-13f41f236987487b8bca287c7ec1fdd3.run_cpu_time_us) (tablet-download-pool-13f41f236987487b8bca287c7ec1fdd3.run_wall_time_us, tablet-download-pool-13f41f236987487b8bca287c7ec1fdd3.run_wall_time_us) (tablet-download-pool-1461e0716aee4f74acefc639cf79eaf0.queue_time_us, tablet-download-pool-1461e0716aee4f74acefc639cf79eaf0.queue_time_us) (tablet-download-pool-1461e0716aee4f74acefc639cf79eaf0.run_cpu_time_us, tablet-download-pool-1461e0716aee4f74acefc639cf79eaf0.run_cpu_time_us) (tablet-download-pool-1461e0716aee4f74acefc639cf79eaf0.run_wall_time_us, tablet-download-pool-1461e0716aee4f74acefc639cf79eaf0.run_wall_time_us) (tablet-download-pool-458ebc1295ee49999586eb699a29bbf8.queue_time_us, tablet-download-pool-458ebc1295ee49999586eb699a29bbf8.queue_time_us) (tablet-download-pool-458ebc1295ee49999586eb699a29bbf8.run_cpu_time_us, tablet-download-pool-458ebc1295ee49999586eb699a29bbf8.run_cpu_time_us) (tablet-download-pool-458ebc1295ee49999586eb699a29bbf8.run_wall_time_us, tablet-download-pool-458ebc1295ee49999586eb699a29bbf8.run_wall_time_us) (tablet-download-pool-58548735201648fa805fdf6af390c088.queue_time_us, tablet-download-pool-58548735201648fa805fdf6af390c088.queue_time_us) (tablet-download-pool-58548735201648fa805fdf6af390c088.run_cpu_time_us, tablet-download-pool-58548735201648fa805fdf6af390c088.run_cpu_time_us) (tablet-download-pool-58548735201648fa805fdf6af390c088.run_wall_time_us, tablet-download-pool-58548735201648fa805fdf6af390c088.run_wall_time_us) (tablet-download-pool-5dc7cefd974149148dcf856a50ea168c.queue_time_us, tablet-download-pool-5dc7cefd974149148dcf856a50ea168c.queue_time_us) (tablet-download-pool-5dc7cefd974149148dcf856a50ea168c.run_cpu_time_us, tablet-download-pool-5dc7cefd974149148dcf856a50ea168c.run_cpu_time_us) (tablet-download-pool-5dc7cefd974149148dcf856a50ea168c.run_wall_time_us, tablet-download-pool-5dc7cefd974149148dcf856a50ea168c.run_wall_time_us) (tablet-download-pool-62a55d0a1d7c4d7aba6635de1994e60b.queue_time_us, tablet-download-pool-62a55d0a1d7c4d7aba6635de1994e60b.queue_time_us) (tablet-download-pool-62a55d0a1d7c4d7aba6635de1994e60b.run_cpu_time_us, tablet-download-pool-62a55d0a1d7c4d7aba6635de1994e60b.run_cpu_time_us) (tablet-download-pool-62a55d0a1d7c4d7aba6635de1994e60b.run_wall_time_us, tablet-download-pool-62a55d0a1d7c4d7aba6635de1994e60b.run_wall_time_us) (tablet-download-pool-62d262e208a84c14bd296e0c44947813.queue_time_us, tablet-download-pool-62d262e208a84c14bd296e0c44947813.queue_time_us) (tablet-download-pool-62d262e208a84c14bd296e0c44947813.run_cpu_time_us, tablet-download-pool-62d262e208a84c14bd296e0c44947813.run_cpu_time_us) (tablet-download-pool-62d262e208a84c14bd296e0c44947813.run_wall_time_us, tablet-download-pool-62d262e208a84c14bd296e0c44947813.run_wall_time_us) (tablet-download-pool-652f2bf72ec9462384afcf9854509767.queue_time_us, tablet-download-pool-652f2bf72ec9462384afcf9854509767.queue_time_us) (tablet-download-pool-652f2bf72ec9462384afcf9854509767.run_cpu_time_us, tablet-download-pool-652f2bf72ec9462384afcf9854509767.run_cpu_time_us) (tablet-download-pool-652f2bf72ec9462384afcf9854509767.run_wall_time_us, tablet-download-pool-652f2bf72ec9462384afcf9854509767.run_wall_time_us) (tablet-download-pool-69009d7fef744e66800ce1f07b4a62bf.queue_time_us, tablet-download-pool-69009d7fef744e66800ce1f07b4a62bf.queue_time_us) (tablet-download-pool-8477be3143f64d83a4c68f260a66efbf.queue_time_us, tablet-download-pool-8477be3143f64d83a4c68f260a66efbf.queue_time_us) (tablet-download-pool-8477be3143f64d83a4c68f260a66efbf.run_cpu_time_us, tablet-download-pool-8477be3143f64d83a4c68f260a66efbf.run_cpu_time_us) (tablet-download-pool-8477be3143f64d83a4c68f260a66efbf.run_wall_time_us, tablet-download-pool-8477be3143f64d83a4c68f260a66efbf.run_wall_time_us) (tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.queue_time_us, tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.queue_time_us) (tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.run_cpu_time_us, tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.run_cpu_time_us) (tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.run_wall_time_us, tablet-download-pool-9b154f780fc742fa9b0efd609c307f95.run_wall_time_us) (tablet-download-pool-e14af3987fbc44e896615d5229d815d7.queue_time_us, tablet-download-pool-e14af3987fbc44e896615d5229d815d7.queue_time_us) (tablet-download-pool-e14af3987fbc44e896615d5229d815d7.run_cpu_time_us, tablet-download-pool-e14af3987fbc44e896615d5229d815d7.run_cpu_time_us) (tablet-download-pool-e14af3987fbc44e896615d5229d815d7.run_wall_time_us, tablet-download-pool-e14af3987fbc44e896615d5229d815d7.run_wall_time_us) (tablet-download-pool-f6f8dfcb870a4461b03415db9c33452e.queue_time_us, tablet-download-pool-f6f8dfcb870a4461b03415db9c33452e.queue_time_us) (tablet-download-pool-f6f8dfcb870a4461b03415db9c33452e.run_cpu_time_us, tablet-download-pool-f6f8dfcb870a4461b03415db9c33452e.run_cpu_time_us) (tablet-download-pool-f6f8dfcb870a4461b03415db9c33452e.run_wall_time_us, tablet-download-pool-f6f8dfcb870a4461b03415db9c33452e.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fd651beddcd google::LogMessage::Fail() at ??:0
@ 0x7fd651bf1b93 google::LogMessage::SendToLog() at ??:0
I20240503 14:13:58.066745 15064 ts_tablet_manager.cc:902] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
@ 0x7fd651bed7cc google::LogMessage::Flush() at ??:0
@ 0x7fd651beef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd65492e742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fd6548faf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fd6548f4cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fd663d488de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 14:13:58.091981 15064 tablet_copy_client.cc:308] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.8:36653
I20240503 14:13:58.092856 9950 tablet_copy_service.cc:140] P ea801601713a4d55aee0de9e3473e0bc: Received BeginTabletCopySession request for tablet 84b58d63957848d7a8b0eae9f60bbbf1 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:44207)
I20240503 14:13:58.093098 9950 tablet_copy_service.cc:161] P ea801601713a4d55aee0de9e3473e0bc: Beginning new tablet copy session on tablet 84b58d63957848d7a8b0eae9f60bbbf1 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:44207: session id = 7186ed8510424fd597173eaf10c57b12-84b58d63957848d7a8b0eae9f60bbbf1
@ 0x7fd663d60daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 14:13:58.097208 9950 tablet_copy_source_session.cc:205] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x7fd663e8b480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 14:13:58.099278 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b58d63957848d7a8b0eae9f60bbbf1. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd663ec3a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 14:13:58.111105 15064 tablet_copy_client.cc:783] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:58.111466 15064 tablet_copy_client.cc:647] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:58.114343 15064 tablet_copy_client.cc:523] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:58.114956 15072 ts_tablet_manager.cc:902] T 122296b00eda44949b9064d0afdd6a16 P 09723f236d8044538d920a37b5d9aa8c: Initiating tablet copy from peer e1bf5fb7402c46829974e2f74cd88708 (127.0.186.4:44337)
@ 0x7fd663ea6981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
I20240503 14:13:58.118897 15064 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
@ 0x7fd663ea670e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fd6548feb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd65490219c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd654902019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fd6548dd01e kudu::Thread::SuperviseThread() at ??:0
W20240503 14:13:58.140506 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
@ 0x7fd657c626db start_thread at ??:0
@ 0x7fd650aa071f clone at ??:0
W20240503 14:13:58.147686 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:58.150213 15064 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:58.150612 15064 tablet_bootstrap.cc:492] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:58.150866 15064 ts_tablet_manager.cc:1367] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.011s
I20240503 14:13:58.151671 15064 raft_consensus.cc:348] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:58.151999 15064 raft_consensus.cc:729] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:58.152144 15064 consensus_queue.cc:260] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:58.156711 15064 ts_tablet_manager.cc:1398] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 14:13:58.162045 9950 tablet_copy_service.cc:342] P ea801601713a4d55aee0de9e3473e0bc: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-84b58d63957848d7a8b0eae9f60bbbf1 received from {username='slave'} at 127.0.186.2:44207
I20240503 14:13:58.162173 9950 tablet_copy_service.cc:433] P ea801601713a4d55aee0de9e3473e0bc: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-84b58d63957848d7a8b0eae9f60bbbf1 on tablet 84b58d63957848d7a8b0eae9f60bbbf1 with peer 7186ed8510424fd597173eaf10c57b12
W20240503 14:13:58.217228 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.334656 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.346871 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.372223 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.391381 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.401973 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.414835 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.421802 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.427006 9735 connection.cc:535] client connection to 127.0.186.3:44691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:58.428459 9469 connection.cc:535] client connection to 127.0.186.3:44691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:58.428619 9865 connection.cc:535] client connection to 127.0.186.3:44691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:58.429422 6736 connection.cc:535] client connection to 127.0.186.3:44691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:58.430600 6867 connection.cc:535] client connection to 127.0.186.3:44691 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:58.432358 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.432474 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:13:58.432507 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.432533 14771 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:13:58.427795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.186.3:44691 (local address 127.0.186.7:34067)
0503 14:13:58.428279 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0503 14:13:58.428297 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0503 14:13:58.428410 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:13:58.431497 (+ 3087us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":359}
W20240503 14:13:58.432608 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.432736 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.432834 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.433609 14760 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:13:58.429129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.186.3:44691 (local address 127.0.186.4:60553)
0503 14:13:58.429391 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0503 14:13:58.429410 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:13:58.429528 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:13:58.432868 (+ 3340us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":158}
W20240503 14:13:58.433815 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 14:13:58.434015 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.434150 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.434233 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.434345 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.434885 14761 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:13:58.430250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.186.3:44691 (local address 127.0.186.1:36151)
0503 14:13:58.430507 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0503 14:13:58.430524 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0503 14:13:58.430629 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:13:58.434187 (+ 3558us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":139}
W20240503 14:13:58.434973 14775 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 14:13:58.429168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.186.3:44691 (local address 127.0.186.5:57391)
0503 14:13:58.429485 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0503 14:13:58.429504 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0503 14:13:58.429614 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 14:13:58.434695 (+ 5081us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":200}
W20240503 14:13:58.435262 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.435950 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.436070 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.436178 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.436273 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.436712 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.440100 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.441468 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:58.441757 15064 ts_tablet_manager.cc:902] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:58.443053 15064 tablet_copy_client.cc:308] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.7:42783
I20240503 14:13:58.443699 9819 tablet_copy_service.cc:140] P 3efdbd3506274dba846ae40404f0572c: Received BeginTabletCopySession request for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:50341)
I20240503 14:13:58.443878 9819 tablet_copy_service.cc:161] P 3efdbd3506274dba846ae40404f0572c: Beginning new tablet copy session on tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:50341: session id = 7186ed8510424fd597173eaf10c57b12-bbbd9f57ea9f457ab0ad4e59bc0f34bf
I20240503 14:13:58.445986 9819 tablet_copy_source_session.cc:205] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:13:58.447299 15064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:13:58.452565 15064 tablet_copy_client.cc:783] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:13:58.453116 15064 tablet_copy_client.cc:647] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:13:58.454579 15064 tablet_copy_client.cc:523] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:13:58.457047 15064 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
I20240503 14:13:58.461266 15064 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:13:58.461570 15064 tablet_bootstrap.cc:492] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:13:58.461762 15064 ts_tablet_manager.cc:1367] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 14:13:58.462345 15064 raft_consensus.cc:348] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:58.462532 15064 raft_consensus.cc:729] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:13:58.462706 15064 consensus_queue.cc:260] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:58.463156 15064 ts_tablet_manager.cc:1398] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 14:13:58.464342 9819 tablet_copy_service.cc:342] P 3efdbd3506274dba846ae40404f0572c: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-bbbd9f57ea9f457ab0ad4e59bc0f34bf received from {username='slave'} at 127.0.186.2:50341
I20240503 14:13:58.464486 9819 tablet_copy_service.cc:433] P 3efdbd3506274dba846ae40404f0572c: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-bbbd9f57ea9f457ab0ad4e59bc0f34bf on tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf with peer 7186ed8510424fd597173eaf10c57b12
W20240503 14:13:58.471550 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.476919 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:13:58.485373 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3f2bd341493482f9519e6dd74cd2f3d: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:58.500998 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.514019 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.516073 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.517699 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.528589 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:58.531828 6929 raft_consensus.cc:1203] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240503 14:13:58.535341 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.539539 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.540891 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.553530 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.563388 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:58.564173 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.603538 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.627492 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:13:58.629859 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.642210 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.646936 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.647558 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:13:58.653082 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.653614 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.656499 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.674860 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.691180 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.708928 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:58.711902 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.739643 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.781719 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:58.931051 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:13:59.006692 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:59.039238 15198 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.039407 15198 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:59.040035 15198 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:13:59.041414 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:13:59.041544 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:59.041663 7125 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:59.041885 15198 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:13:59.046725 14354 raft_consensus.cc:1050] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:59.047472 14354 consensus_queue.cc:237] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:59.050060 6929 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.050122 7191 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.050297 9532 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 14:13:59.050588 15202 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:59.051021 15145 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:59.051332 14645 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 14:13:59.056141 15202 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.056851 6929 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.057560 7191 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.061599 9530 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.063908 6654 catalog_manager.cc:5484] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: config changed from index 5 to 6, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.065249 6929 raft_consensus.cc:1203] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 14:13:59.073206 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.083946 9928 consensus_queue.cc:237] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:59.087549 6929 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:59.091452 7191 raft_consensus.cc:1261] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Refusing update from remote peer ea801601713a4d55aee0de9e3473e0bc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 14:13:59.091841 15145 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.092855 15145 consensus_queue.cc:1035] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.095166 14645 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P ea801601713a4d55aee0de9e3473e0bc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.095809 7191 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.101219 6644 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84b58d63957848d7a8b0eae9f60bbbf1 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:13:59.102710 6654 catalog_manager.cc:5484] T 84b58d63957848d7a8b0eae9f60bbbf1 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 6 to 7, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New cstate: current_term: 1 leader_uuid: "ea801601713a4d55aee0de9e3473e0bc" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.104964 6929 raft_consensus.cc:2932] T 84b58d63957848d7a8b0eae9f60bbbf1 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.117861 9513 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b58d63957848d7a8b0eae9f60bbbf1 with delete_type TABLET_DATA_TOMBSTONED (TS 5b7e36bf354f480c90859cb8867b59cf not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33196
W20240503 14:13:59.124006 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:59.131822 15215 tablet_replica.cc:330] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: stopping tablet replica
I20240503 14:13:59.132176 15215 raft_consensus.cc:2229] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 14:13:59.132486 15215 raft_consensus.cc:2258] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 14:13:59.133569 15215 ts_tablet_manager.cc:1875] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 14:13:59.141737 15215 ts_tablet_manager.cc:1888] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 14:13:59.141925 15215 log.cc:1196] T 84b58d63957848d7a8b0eae9f60bbbf1 P 5b7e36bf354f480c90859cb8867b59cf: Deleting WAL directory at /tmp/dist-test-tasknKJaW5/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714745514914807-744-0/raft_consensus-itest-cluster/ts-4/wal/wals/84b58d63957848d7a8b0eae9f60bbbf1
I20240503 14:13:59.143039 6646 catalog_manager.cc:4830] TS 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609): tablet 84b58d63957848d7a8b0eae9f60bbbf1 (table rebalancer_extra_table_16 [id=eb8672479c7e409ea735864d89e0c79c]) successfully deleted
W20240503 14:13:59.161350 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.176381 15200 raft_consensus.cc:482] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.176590 15200 raft_consensus.cc:504] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:59.177361 14045 raft_consensus.cc:482] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.177428 15200 leader_election.cc:288] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:59.177500 14045 raft_consensus.cc:504] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:59.178063 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a755b35d694a2b89ca1f58f0fa7976" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:59.178216 14045 leader_election.cc:288] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:59.178994 15182 raft_consensus.cc:482] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.179137 15182 raft_consensus.cc:504] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:59.180795 15182 leader_election.cc:288] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:59.182140 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90c8e47862945f1a873b17da14bb397" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
W20240503 14:13:59.183530 9469 leader_election.cc:334] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.183722 9469 leader_election.cc:302] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13
W20240503 14:13:59.184239 6736 leader_election.cc:334] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.184489 15200 raft_consensus.cc:2726] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:13:59.185505 9735 leader_election.cc:334] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.185665 9735 leader_election.cc:302] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:59.185874 14045 raft_consensus.cc:2726] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:59.186308 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e68127f4f94413b1be984a87bec6a8" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:59.186844 6736 leader_election.cc:302] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f1a669e28f014fe287932acf56bebc13; no voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:59.187325 15182 raft_consensus.cc:2726] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:59.221607 14725 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.221845 14725 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:13:59.222802 14725 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:13:59.225471 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:59.229187 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.229322 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:59.229825 14725 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:13:59.240908 15182 raft_consensus.cc:482] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.241133 15182 raft_consensus.cc:504] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:59.241784 15182 leader_election.cc:288] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:59.243669 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90c8e47862945f1a873b17da14bb397" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:59.243898 9799 raft_consensus.cc:2454] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:59.244346 6735 leader_election.cc:302] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:59.244740 15182 raft_consensus.cc:2781] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:59.244863 15182 raft_consensus.cc:482] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.244964 15182 raft_consensus.cc:3037] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.247761 15182 raft_consensus.cc:504] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
W20240503 14:13:59.248513 6736 leader_election.cc:334] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.248873 15157 raft_consensus.cc:482] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.249634 15157 raft_consensus.cc:504] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:59.251158 15182 leader_election.cc:288] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:59.251161 15157 leader_election.cc:288] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:59.252089 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a90c8e47862945f1a873b17da14bb397" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:59.252344 9799 raft_consensus.cc:3037] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.254956 9799 raft_consensus.cc:2454] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:59.256635 6735 leader_election.cc:302] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:59.256985 15182 raft_consensus.cc:2781] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:59.257208 15182 raft_consensus.cc:686] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [term 2 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
W20240503 14:13:59.257499 9735 leader_election.cc:334] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.257571 15182 consensus_queue.cc:237] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:59.258096 14645 raft_consensus.cc:482] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.258283 14645 raft_consensus.cc:504] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:59.259047 14645 leader_election.cc:288] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285)
I20240503 14:13:59.260316 6929 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca522e5b44fd44289b4b161f27f71d20" candidate_uuid: "ea801601713a4d55aee0de9e3473e0bc" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7186ed8510424fd597173eaf10c57b12" is_pre_election: true
W20240503 14:13:59.261579 9865 leader_election.cc:334] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.261893 9863 leader_election.cc:302] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea801601713a4d55aee0de9e3473e0bc; no voters: 09723f236d8044538d920a37b5d9aa8c, 7186ed8510424fd597173eaf10c57b12
I20240503 14:13:59.262286 14645 raft_consensus.cc:2726] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:13:59.263044 6736 leader_election.cc:334] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.265623 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59daa2d12f446899c7bb9970e1694da" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:59.266494 9733 leader_election.cc:302] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f1a669e28f014fe287932acf56bebc13
I20240503 14:13:59.266991 15157 raft_consensus.cc:2726] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:59.269945 6658 catalog_manager.cc:5484] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 2 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:59.281379 15157 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.281637 15157 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:59.282464 15157 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:13:59.285573 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:13:59.293267 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:59.293437 9733 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:59.293762 15157 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:13:59.303059 15182 raft_consensus.cc:482] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.303280 15182 raft_consensus.cc:504] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:59.304051 15182 leader_election.cc:288] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:59.304868 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59daa2d12f446899c7bb9970e1694da" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:59.305117 9799 raft_consensus.cc:2454] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:59.305761 6735 leader_election.cc:302] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:59.306064 15182 raft_consensus.cc:2781] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:59.306159 15182 raft_consensus.cc:482] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.306391 15182 raft_consensus.cc:3037] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.309343 15182 raft_consensus.cc:504] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:59.311673 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59daa2d12f446899c7bb9970e1694da" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:59.311975 9799 raft_consensus.cc:3037] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.315353 9799 raft_consensus.cc:2454] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:59.316565 15182 leader_election.cc:288] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Requested vote from peers 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:13:59.317015 6736 leader_election.cc:334] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.318754 6735 leader_election.cc:302] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, f1a669e28f014fe287932acf56bebc13; no voters:
W20240503 14:13:59.319550 6736 leader_election.cc:334] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.320484 15182 raft_consensus.cc:2781] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:59.320824 15182 raft_consensus.cc:686] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [term 2 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:59.323231 15182 consensus_queue.cc:237] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } }
I20240503 14:13:59.327668 15157 raft_consensus.cc:482] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.327829 15157 raft_consensus.cc:504] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:59.328522 15157 leader_election.cc:288] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
I20240503 14:13:59.329386 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "3efdbd3506274dba846ae40404f0572c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:59.335458 6658 catalog_manager.cc:5484] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 2 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:13:59.342478 9735 leader_election.cc:334] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.342649 9735 leader_election.cc:302] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, 5b7e36bf354f480c90859cb8867b59cf
I20240503 14:13:59.343178 15157 raft_consensus.cc:2726] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:13:59.377152 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.415536 14624 raft_consensus.cc:482] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.415760 14624 raft_consensus.cc:504] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:59.416469 14624 leader_election.cc:288] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:59.418639 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a755b35d694a2b89ca1f58f0fa7976" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf" is_pre_election: true
I20240503 14:13:59.418911 9530 raft_consensus.cc:2454] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 1.
I20240503 14:13:59.420367 6736 leader_election.cc:302] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:59.420866 15182 raft_consensus.cc:2781] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:59.420943 15182 raft_consensus.cc:482] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.421013 15182 raft_consensus.cc:3037] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.423571 15182 raft_consensus.cc:504] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
W20240503 14:13:59.425505 6736 leader_election.cc:334] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.426081 15182 leader_election.cc:288] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:13:59.426728 9530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a755b35d694a2b89ca1f58f0fa7976" candidate_uuid: "f1a669e28f014fe287932acf56bebc13" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5b7e36bf354f480c90859cb8867b59cf"
I20240503 14:13:59.426995 9530 raft_consensus.cc:3037] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:59.428030 6736 leader_election.cc:334] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.430042 9530 raft_consensus.cc:2454] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1a669e28f014fe287932acf56bebc13 in term 2.
I20240503 14:13:59.431445 6736 leader_election.cc:302] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:59.431725 15182 raft_consensus.cc:2781] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:59.431860 15182 raft_consensus.cc:686] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [term 2 LEADER]: Becoming Leader. State: Replica: f1a669e28f014fe287932acf56bebc13, State: Running, Role: LEADER
I20240503 14:13:59.432129 15182 consensus_queue.cc:237] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
W20240503 14:13:59.433156 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:59.445571 6658 catalog_manager.cc:5484] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to f1a669e28f014fe287932acf56bebc13 (127.0.186.1). New cstate: current_term: 2 leader_uuid: "f1a669e28f014fe287932acf56bebc13" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:59.490242 15199 raft_consensus.cc:482] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.490425 15199 raft_consensus.cc:504] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:59.491134 15199 leader_election.cc:288] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:59.491901 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c" is_pre_election: true
I20240503 14:13:59.492136 9799 raft_consensus.cc:2454] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:59.492525 9468 leader_election.cc:302] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters:
I20240503 14:13:59.492763 15199 raft_consensus.cc:2781] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:59.492878 15199 raft_consensus.cc:482] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.492969 15199 raft_consensus.cc:3037] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:59.493340 9469 leader_election.cc:334] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.495869 15199 raft_consensus.cc:504] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:59.496949 15199 leader_election.cc:288] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), 3efdbd3506274dba846ae40404f0572c (127.0.186.7:42783)
I20240503 14:13:59.497045 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5895d998bfb3439386fcbc2c74f8d9a6" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "3efdbd3506274dba846ae40404f0572c"
I20240503 14:13:59.497233 9799 raft_consensus.cc:3037] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:59.497931 9469 leader_election.cc:334] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.499380 9799 raft_consensus.cc:2454] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
I20240503 14:13:59.512135 9468 leader_election.cc:302] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c, 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:59.512698 15199 raft_consensus.cc:2781] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:59.512873 15199 raft_consensus.cc:686] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:59.513136 15199 consensus_queue.cc:237] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:13:59.514982 15200 raft_consensus.cc:482] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.515175 15200 raft_consensus.cc:504] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:59.515964 15200 leader_election.cc:288] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:59.517422 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e68127f4f94413b1be984a87bec6a8" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13" is_pre_election: true
I20240503 14:13:59.517660 6800 raft_consensus.cc:2454] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 1.
I20240503 14:13:59.518138 9467 leader_election.cc:302] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters:
I20240503 14:13:59.518595 15199 raft_consensus.cc:2781] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:59.518692 15199 raft_consensus.cc:482] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.518774 15199 raft_consensus.cc:3037] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:59.520627 9469 leader_election.cc:334] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.521538 15199 raft_consensus.cc:504] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:59.522239 6658 catalog_manager.cc:5484] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 14:13:59.522663 15199 leader_election.cc:288] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f1a669e28f014fe287932acf56bebc13 (127.0.186.1:34101)
I20240503 14:13:59.524956 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21e68127f4f94413b1be984a87bec6a8" candidate_uuid: "5b7e36bf354f480c90859cb8867b59cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f1a669e28f014fe287932acf56bebc13"
I20240503 14:13:59.525190 6800 raft_consensus.cc:3037] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 1 FOLLOWER]: Advancing to term 2
W20240503 14:13:59.526791 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3f2bd341493482f9519e6dd74cd2f3d: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:59.527813 6800 raft_consensus.cc:2454] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b7e36bf354f480c90859cb8867b59cf in term 2.
W20240503 14:13:59.528682 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:59.530575 9469 leader_election.cc:334] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.531428 9467 leader_election.cc:302] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf, f1a669e28f014fe287932acf56bebc13; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:59.531915 15199 raft_consensus.cc:2781] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:59.532045 15199 raft_consensus.cc:686] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Becoming Leader. State: Replica: 5b7e36bf354f480c90859cb8867b59cf, State: Running, Role: LEADER
I20240503 14:13:59.532282 15199 consensus_queue.cc:237] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } }
I20240503 14:13:59.538053 6658 catalog_manager.cc:5484] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5). New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:59.545572 15201 raft_consensus.cc:482] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.545781 15201 raft_consensus.cc:504] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:59.546451 15201 leader_election.cc:288] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
W20240503 14:13:59.549466 6867 leader_election.cc:334] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.549593 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca522e5b44fd44289b4b161f27f71d20" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc" is_pre_election: true
I20240503 14:13:59.549824 9928 raft_consensus.cc:2454] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 1.
I20240503 14:13:59.550633 6865 leader_election.cc:302] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:59.551103 15201 raft_consensus.cc:2781] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 14:13:59.551188 15201 raft_consensus.cc:482] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.551255 15201 raft_consensus.cc:3037] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.553653 15201 raft_consensus.cc:504] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
W20240503 14:13:59.556624 6867 leader_election.cc:334] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.557181 15201 leader_election.cc:288] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 election: Requested vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), ea801601713a4d55aee0de9e3473e0bc (127.0.186.8:36653)
I20240503 14:13:59.557837 9928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca522e5b44fd44289b4b161f27f71d20" candidate_uuid: "7186ed8510424fd597173eaf10c57b12" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "ea801601713a4d55aee0de9e3473e0bc"
I20240503 14:13:59.558055 9928 raft_consensus.cc:3037] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Advancing to term 2
I20240503 14:13:59.561538 9928 raft_consensus.cc:2454] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7186ed8510424fd597173eaf10c57b12 in term 2.
W20240503 14:13:59.562011 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:59.562512 6865 leader_election.cc:302] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7186ed8510424fd597173eaf10c57b12, ea801601713a4d55aee0de9e3473e0bc; no voters: 09723f236d8044538d920a37b5d9aa8c
I20240503 14:13:59.562885 15201 raft_consensus.cc:2781] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 14:13:59.563100 15201 raft_consensus.cc:686] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [term 2 LEADER]: Becoming Leader. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Running, Role: LEADER
I20240503 14:13:59.563397 15201 consensus_queue.cc:237] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } }
I20240503 14:13:59.567804 6658 catalog_manager.cc:5484] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 reported cstate change: term changed from 1 to 2, leader changed from 09723f236d8044538d920a37b5d9aa8c (127.0.186.3) to 7186ed8510424fd597173eaf10c57b12 (127.0.186.2). New cstate: current_term: 2 leader_uuid: "7186ed8510424fd597173eaf10c57b12" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 14:13:59.572155 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:59.573797 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:13:59.579420 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.588593 15199 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:13:59.588766 15199 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:13:59.589448 15199 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
I20240503 14:13:59.589655 15157 raft_consensus.cc:1050] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:13:59.590433 15157 consensus_queue.cc:237] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
W20240503 14:13:59.591178 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.602167 6929 raft_consensus.cc:1261] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12 [term 1 LEARNER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:59.602434 9928 raft_consensus.cc:1261] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 14:13:59.603019 15323 consensus_queue.cc:1035] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 14:13:59.603245 14045 consensus_queue.cc:1035] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 14:13:59.603932 9733 consensus_peers.cc:485] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:59.604399 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:13:59.604526 9467 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:13:59.604776 15199 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:13:59.608387 14045 raft_consensus.cc:2932] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.612226 6929 raft_consensus.cc:2932] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.614709 9928 raft_consensus.cc:2932] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.615398 6658 catalog_manager.cc:5484] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 2 to 3, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:59.622249 9799 consensus_queue.cc:237] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:13:59.628307 6929 raft_consensus.cc:1261] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 14:13:59.629165 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:59.629465 14779 consensus_queue.cc:1035] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:59.633347 6929 raft_consensus.cc:2932] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 7186ed8510424fd597173eaf10c57b12 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.633452 15157 raft_consensus.cc:2932] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:13:59.636595 9929 raft_consensus.cc:1261] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Refusing update from remote peer 3efdbd3506274dba846ae40404f0572c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 14:13:59.637526 14779 consensus_queue.cc:1035] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 14:13:59.638957 9929 raft_consensus.cc:2932] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P ea801601713a4d55aee0de9e3473e0bc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
W20240503 14:13:59.644356 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:59.644070 6658 catalog_manager.cc:5484] T bbbd9f57ea9f457ab0ad4e59bc0f34bf P 3efdbd3506274dba846ae40404f0572c reported cstate change: config changed from index 3 to 4, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 1 leader_uuid: "3efdbd3506274dba846ae40404f0572c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 14:13:59.645817 6645 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 14:13:59.728086 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.816771 9799 raft_consensus.cc:1261] T c59daa2d12f446899c7bb9970e1694da P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:59.817394 14624 consensus_queue.cc:1035] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:59.830214 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:59.855262 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:13:59.859547 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:59.867695 9467 consensus_peers.cc:485] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:13:59.880477 9532 raft_consensus.cc:1261] T 09a755b35d694a2b89ca1f58f0fa7976 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:59.881039 15340 consensus_queue.cc:1035] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:13:59.881870 9799 raft_consensus.cc:1261] T a90c8e47862945f1a873b17da14bb397 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer f1a669e28f014fe287932acf56bebc13: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:59.882571 15340 consensus_queue.cc:1035] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:13:59.907996 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.924908 9799 raft_consensus.cc:1261] T 5895d998bfb3439386fcbc2c74f8d9a6 P 3efdbd3506274dba846ae40404f0572c [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 14:13:59.925474 14660 consensus_queue.cc:1035] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 14:13:59.934516 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.951259 6800 raft_consensus.cc:1261] T 21e68127f4f94413b1be984a87bec6a8 P f1a669e28f014fe287932acf56bebc13 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:13:59.951970 15200 consensus_queue.cc:1035] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a669e28f014fe287932acf56bebc13" member_type: VOTER last_known_addr { host: "127.0.186.1" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:13:59.960566 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:59.979360 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:13:59.986315 9532 consensus_queue.cc:237] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:13:59.988752 7191 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 14:13:59.989391 14660 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 14:13:59.989645 9467 consensus_peers.cc:485] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 7186ed8510424fd597173eaf10c57b12 (127.0.186.2:45285): Couldn't send request to peer 7186ed8510424fd597173eaf10c57b12. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 603002fdf6b84d3ea63c65636eb42cb1. This is attempt 1: this message will repeat every 5th retry.
W20240503 14:13:59.989965 9467 consensus_peers.cc:485] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:13:59.991909 15200 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:59.992646 7191 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
I20240503 14:13:59.998287 6654 catalog_manager.cc:5484] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 reported cstate change: config changed from index 4 to 6, NON_VOTER 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) added. New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } } }
W20240503 14:14:00.044682 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:00.045563 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.079680 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:00.104760 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:00.106781 9928 raft_consensus.cc:1261] T ca522e5b44fd44289b4b161f27f71d20 P ea801601713a4d55aee0de9e3473e0bc [term 2 FOLLOWER]: Refusing update from remote peer 7186ed8510424fd597173eaf10c57b12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:00.107939 14379 consensus_queue.cc:1035] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea801601713a4d55aee0de9e3473e0bc" member_type: VOTER last_known_addr { host: "127.0.186.8" port: 36653 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:14:00.184652 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:00.202453 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.238688 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.245906 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.383364 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.410113 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.467723 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:00.482030 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:00.489753 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.491299 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.544095 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:14:00.553092 15485 ts_tablet_manager.cc:902] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: Initiating tablet copy from peer 5b7e36bf354f480c90859cb8867b59cf (127.0.186.5:36609)
I20240503 14:14:00.554708 15485 tablet_copy_client.cc:308] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Beginning tablet copy session from remote peer at address 127.0.186.5:36609
I20240503 14:14:00.555574 9553 tablet_copy_service.cc:140] P 5b7e36bf354f480c90859cb8867b59cf: Received BeginTabletCopySession request for tablet 603002fdf6b84d3ea63c65636eb42cb1 from peer 7186ed8510424fd597173eaf10c57b12 ({username='slave'} at 127.0.186.2:57189)
I20240503 14:14:00.555769 9553 tablet_copy_service.cc:161] P 5b7e36bf354f480c90859cb8867b59cf: Beginning new tablet copy session on tablet 603002fdf6b84d3ea63c65636eb42cb1 from peer 7186ed8510424fd597173eaf10c57b12 at {username='slave'} at 127.0.186.2:57189: session id = 7186ed8510424fd597173eaf10c57b12-603002fdf6b84d3ea63c65636eb42cb1
I20240503 14:14:00.558099 9553 tablet_copy_source_session.cc:205] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 14:14:00.560127 15485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 603002fdf6b84d3ea63c65636eb42cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 14:14:00.565322 15485 tablet_copy_client.cc:783] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 0 data blocks...
I20240503 14:14:00.565564 15485 tablet_copy_client.cc:647] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Starting download of 1 WAL segments...
I20240503 14:14:00.567234 15485 tablet_copy_client.cc:523] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 14:14:00.569797 15485 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: Bootstrap starting.
W20240503 14:14:00.572654 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:00.576931 15485 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 14:14:00.577235 15485 tablet_bootstrap.cc:492] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: Bootstrap complete.
I20240503 14:14:00.577443 15485 ts_tablet_manager.cc:1367] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.001s
I20240503 14:14:00.578230 15485 raft_consensus.cc:348] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:14:00.578512 15485 raft_consensus.cc:729] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7186ed8510424fd597173eaf10c57b12, State: Initialized, Role: LEARNER
I20240503 14:14:00.578733 15485 consensus_queue.cc:260] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: NON_VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: true } }
I20240503 14:14:00.579440 15485 ts_tablet_manager.cc:1398] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 14:14:00.580225 9553 tablet_copy_service.cc:342] P 5b7e36bf354f480c90859cb8867b59cf: Request end of tablet copy session 7186ed8510424fd597173eaf10c57b12-603002fdf6b84d3ea63c65636eb42cb1 received from {username='slave'} at 127.0.186.2:57189
I20240503 14:14:00.580346 9553 tablet_copy_service.cc:433] P 5b7e36bf354f480c90859cb8867b59cf: ending tablet copy session 7186ed8510424fd597173eaf10c57b12-603002fdf6b84d3ea63c65636eb42cb1 on tablet 603002fdf6b84d3ea63c65636eb42cb1 with peer 7186ed8510424fd597173eaf10c57b12
W20240503 14:14:00.607232 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:14:00.649713 14725 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:00.649921 14725 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:00.650626 14725 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:00.652575 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:00.655189 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:00.655310 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:00.655853 14724 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:00.660035 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.681881 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.697937 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:00.707428 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.761689 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.777966 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.778681 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.791493 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.800971 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.803943 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.815110 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.820276 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:00.839150 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.844655 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.865993 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.871582 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.882135 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.882659 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.883674 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.894261 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:00.916790 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.926515 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240503 14:14:00.927990 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 14:14:00.928934 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.933943 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:14:00.934813 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:14:00.945079 6928 raft_consensus.cc:1203] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240503 14:14:00.949335 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240503 14:14:00.950148 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:00.969786 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.973421 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.982367 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:00.988688 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240503 14:14:00.995805 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:00.999827 9467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:14:01.011139 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.017340 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.043164 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.070048 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.081228 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.084419 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.087437 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.118626 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.129704 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.147548 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.149448 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.150791 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.163508 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.167143 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:01.190651 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.206108 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.238811 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:01.253827 15511 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:01.254006 15511 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:01.256356 15511 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:01.256879 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:01.258011 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:01.258112 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:01.258330 15511 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:01.270467 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.287216 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.314874 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.324946 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.327759 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:14:01.350382 15541 raft_consensus.cc:1050] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf: attempting to promote NON_VOTER 7186ed8510424fd597173eaf10c57b12 to VOTER
I20240503 14:14:01.351202 15541 consensus_queue.cc:237] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:14:01.353833 7190 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:01.353960 6928 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12 [term 2 LEARNER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 14:14:01.355237 15484 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 14:14:01.355746 15484 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 14:14:01.359542 9467 consensus_peers.cc:485] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 14:14:01.363809 15199 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
W20240503 14:14:01.370127 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 14:14:01.370657 7190 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:14:01.374182 6652 catalog_manager.cc:5484] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 6 to 7, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:14:01.385560 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:14:01.385326 6928 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 7186ed8510424fd597173eaf10c57b12 (127.0.186.2) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { replace: true } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:14:01.403055 9530 consensus_queue.cc:237] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } }
I20240503 14:14:01.406491 6928 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:01.406652 7190 raft_consensus.cc:1261] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Refusing update from remote peer 5b7e36bf354f480c90859cb8867b59cf: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 14:14:01.407104 15199 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.407544 15199 consensus_queue.cc:1035] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 14:14:01.410332 15625 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:14:01.410931 7190 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:14:01.413620 6646 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 603002fdf6b84d3ea63c65636eb42cb1 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 14:14:01.414924 6928 raft_consensus.cc:2932] T 603002fdf6b84d3ea63c65636eb42cb1 P 7186ed8510424fd597173eaf10c57b12 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } } }
I20240503 14:14:01.420223 6652 catalog_manager.cc:5484] T 603002fdf6b84d3ea63c65636eb42cb1 P 5b7e36bf354f480c90859cb8867b59cf reported cstate change: config changed from index 7 to 8, VOTER f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6) evicted. New cstate: current_term: 2 leader_uuid: "5b7e36bf354f480c90859cb8867b59cf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7186ed8510424fd597173eaf10c57b12" member_type: VOTER last_known_addr { host: "127.0.186.2" port: 45285 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 14:14:01.434967 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:01.468155 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:01.473392 15323 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:01.473635 15323 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:14:01.474462 15323 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:01.476251 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:01.476548 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:01.476635 9733 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:01.476837 15323 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:01.483816 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:01.566588 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:01.642361 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:14:01.704173 15199 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:01.704349 15199 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
W20240503 14:14:01.704582 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:01.705017 15199 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:01.706971 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:01.707420 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:01.707571 9469 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:01.708030 15199 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:01.855722 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:01.876216 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:01.888095 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:01.900729 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:01.949684 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.001122 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.085443 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.114939 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.125838 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:02.183223 15511 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:02.183423 15511 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:02.184206 15511 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:02.186101 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:02.186321 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:02.186414 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:02.186762 15511 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:02.209003 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:02.256002 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.263886 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:02.267442 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:02.286698 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.297083 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:02.320772 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.382481 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:02.462127 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:02.527153 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:02.543351 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:02.544252 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:02.605718 6644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240503 14:14:02.610822 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 14:14:02.623339 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:02.671037 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.755949 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:02.767621 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:02.794754 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:02.828824 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:02.869800 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:02.889865 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:02.937263 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:02.964205 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.008699 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.070128 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.082006 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.095958 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.142027 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:03.149292 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.178385 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:03.209934 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.253249 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.273869 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.274006 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.276263 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.276649 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.317659 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.327569 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:03.339314 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.339429 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.353863 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.379043 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.380151 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:03.407902 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.409906 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.411841 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.412671 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.418541 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.444705 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.462419 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.470227 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.475278 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.485597 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.523046 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.570101 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.580310 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.584160 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.586788 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.588029 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.601015 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:03.613536 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.619252 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.621228 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.629496 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.630332 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:03.636581 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.641371 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 603002fdf6b84d3ea63c65636eb42cb1: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:03.663436 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.666132 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.717347 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.717542 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.734930 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.737095 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:03.758046 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:03.760134 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.760890 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:03.839975 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:03.870993 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240503 14:14:03.880398 15199 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:03.880594 15199 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:14:03.881263 15199 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:03.882748 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:03.883009 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:03.883116 9467 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:03.883363 15199 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 14:14:03.980788 14724 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:03.981046 14724 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:03.981710 14724 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:03.983548 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:03.984556 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:03.984664 7125 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:03.984894 14724 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:04.007140 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.034554 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:04.067299 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:04.077122 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.094290 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:04.122135 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:04.205216 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:04.305169 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:04.318958 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.556972 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:04.569023 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:04.598073 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.598846 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:04.620357 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.629541 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.629983 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.636960 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.649372 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:04.653107 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:04.741896 15323 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:04.742120 15323 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:14:04.742800 15323 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:04.751438 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:04.757544 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:04.766166 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:04.766343 9733 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:04.766592 15323 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:04.852371 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:04.861285 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:04.901302 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:04.903664 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:04.927805 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:05.035533 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:05.042984 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.062515 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:05.075250 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.122411 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 14:14:05.199023 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:05.220317 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.242853 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.340241 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:05.360567 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.417454 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.506225 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:14:05.519021 14724 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:05.519281 14724 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:05.520007 14724 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:05.521780 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:05.522217 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:05.522358 7125 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:05.522607 14724 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:05.530436 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.578935 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:05.610227 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.614215 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.655342 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.664983 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.675956 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:05.680200 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.683761 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:05.689169 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.702756 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.709933 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.715876 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 603002fdf6b84d3ea63c65636eb42cb1: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:05.729080 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.756819 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.760623 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.779641 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.780043 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.815685 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.834975 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.864620 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.879309 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.881983 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.896065 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.908102 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:05.962495 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.980425 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 14:14:05.981595 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.982896 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:05.985682 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:14:05.987768 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:05.988983 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:14:05.989962 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.010775 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 147 similar messages]
W20240503 14:14:06.013674 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 14:14:06.014761 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.021621 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.022562 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 14:14:06.028780 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.061498 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:06.065012 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.080672 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.099267 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.107234 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.133646 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.144074 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.160141 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.183733 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.200891 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:06.208202 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.228039 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.230044 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.274001 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.299227 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:06.311081 15511 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:06.311270 15511 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:06.311941 15511 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:06.313248 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:06.313438 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:06.313539 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:06.313880 15511 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:06.319679 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.330749 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:06.367437 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.383205 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.383826 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.421835 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240503 14:14:06.426856 15323 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:06.427047 15323 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:14:06.427737 15323 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:06.429380 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:06.430146 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:06.430279 9735 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:06.430538 15323 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 14:14:06.436725 16197 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:06.436906 16197 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:14:06.437618 16197 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:06.439386 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:06.439743 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:06.439868 9469 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:06.440105 16197 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:06.457702 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.507843 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:06.531303 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:06.578569 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:06.643406 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.770947 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:06.789757 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.809574 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.866117 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.873993 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.924754 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:06.929678 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:06.963256 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:07.005051 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:07.051101 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:07.077296 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:07.119858 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:07.126196 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:07.140982 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:07.174801 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:07.286855 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:07.294685 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:07.329632 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:07.399040 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:07.408491 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:07.471248 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:07.503701 15511 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:07.503881 15511 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:07.504585 15511 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:07.507136 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:07.507325 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:07.507402 7125 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:07.507692 14724 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:07.569953 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:07.600812 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 14:14:07.610044 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:07.610632 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:07.787107 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:07.787204 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:07.813056 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:07.828624 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:07.852394 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:07.942497 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:07.972692 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:07.996832 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:08.024039 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.045135 6644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 5 similar messages]
W20240503 14:14:08.047223 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:08.047730 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.073963 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.082849 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:08.084672 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.092511 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.106917 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.112463 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.112601 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.133045 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.141813 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:08.146126 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.172683 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.209650 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.210098 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.210669 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.233239 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.271322 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.278859 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.282925 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.294822 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.320451 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.343205 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.353924 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.368959 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.379547 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:08.446735 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.455840 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:08.468410 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.495323 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.516798 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.521481 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.523159 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.531273 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.555086 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.561321 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.574259 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.583671 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.637513 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.666810 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:08.685473 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.705947 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240503 14:14:08.711897 16578 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:08.712097 16578 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:14:08.713033 16578 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:08.714583 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:08.714942 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:08.715291 9733 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:08.715608 16578 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:08.718700 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.733098 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:08.768944 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.792678 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.798172 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.830991 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.880337 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.903771 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:08.904325 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:08.929154 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:08.944311 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:08.952668 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:09.151384 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240503 14:14:09.153225 16620 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:09.153395 16620 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:09.154099 16620 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:09.155340 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:09.155954 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:09.156045 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:09.156314 16620 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:09.172859 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:09.256297 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:09.276082 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:09.289357 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.305513 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:09.385301 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:09.398481 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.445636 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:09.478693 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.503643 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:09.503731 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.521459 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.532011 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:09.579639 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.635650 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.663969 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.687130 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:09.839839 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:09.841739 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:09.870831 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 603002fdf6b84d3ea63c65636eb42cb1: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:09.895541 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:09.908315 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:09.950544 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:09.951717 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:09.990168 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:10.035864 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.064880 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:10.093079 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 14:14:10.160077 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:10.218232 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.256734 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.271330 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:10.320487 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.380702 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.433782 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.438226 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.453982 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.477171 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:10.480533 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.500283 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.507419 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:10.510123 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.520062 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.522962 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.563170 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.571990 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.634169 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.646100 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.652952 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.684194 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:10.705539 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.738770 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.763662 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.766111 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:10.788903 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.798483 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.804948 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.844465 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.859143 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.866549 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.951452 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.960474 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.964946 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.967048 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.969656 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:10.977191 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.981259 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:10.991197 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 14:14:11.001099 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:11.049336 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:14:11.050313 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.054889 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 14:14:11.056053 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.056396 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:14:11.060061 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:11.080374 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.091527 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 14:14:11.093854 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.101857 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.102933 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.130376 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:14:11.140864 16878 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:11.141069 16878 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:14:11.141803 16878 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:11.144009 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240503 14:14:11.144901 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:11.146782 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:11.146916 9467 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:11.147356 16878 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:11.185248 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.202805 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.268860 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.278765 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.278785 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:11.298564 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.307864 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:11.328001 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.340916 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.361694 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.383623 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:11.458865 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.492863 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.523836 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.563024 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.569662 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.648361 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:11.728286 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:11.749828 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.755312 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.778797 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:11.783857 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.854416 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.902242 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:11.923394 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.942159 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:11.977548 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:12.056728 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.097136 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.142855 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.227524 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:12.240641 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:12.270491 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:12.331470 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:12.344414 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.350620 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.382758 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:12.416792 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:12.454257 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:12.480826 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:12.596051 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.607939 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:12.615377 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:12.680524 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.688774 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240503 14:14:12.740029 17089 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:12.740217 17089 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:12.740842 17089 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:12.744385 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:12.745674 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:12.746325 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:12.746912 17089 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:12.746557 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:12.771063 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 14:14:12.809288 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.828701 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.832129 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:12.900128 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:12.987179 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:12.998407 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:13.005882 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:13.017825 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.090759 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.121815 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.129827 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.144240 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.194267 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.212620 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.218677 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.226874 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.233944 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.249388 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.261174 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:13.261266 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.286437 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.320792 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.321873 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.327153 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.335006 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.362088 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.362670 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.375880 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:13.404635 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:13.426784 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.430477 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.444026 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:13.478528 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.480376 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.520900 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.551501 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.563331 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.589957 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.593536 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.620182 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.620358 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.632238 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.670118 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.703467 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.716449 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:13.765143 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.794526 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.800158 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.805167 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.805660 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:13.838203 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.838549 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.885212 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:13.915541 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.923996 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:13.940994 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:13.948822 6644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240503 14:14:13.949992 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3f2bd341493482f9519e6dd74cd2f3d: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:13.998288 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:13.998637 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:14.046191 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.105906 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.112125 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:14.149603 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:14.155618 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.202903 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:14.244240 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:14.246713 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.366114 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:14.400741 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:14.454890 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:14.464957 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.475162 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:14.479393 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:14.555416 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.561715 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.586426 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:14.632092 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.651716 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.685622 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:14.793237 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:14.825312 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:14.857234 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:14.892002 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:14.923424 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:14.923424 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:14.951951 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:15.018517 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.027469 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.086272 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:15.278770 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:15.305807 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.324427 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.328646 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.418859 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:15.453588 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:15.458689 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.463682 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.477523 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 14:14:15.506893 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.511694 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:15.517313 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.535025 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.539680 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.549036 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.573277 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.607439 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240503 14:14:15.630288 17415 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:15.630491 17415 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:15.631161 17415 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:15.632690 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:15.634070 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:15.634197 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:15.635017 17553 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:15.686790 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.692515 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.748878 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.753968 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.766466 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.767688 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.770543 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.775409 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:15.781459 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.839804 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:15.866750 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.873601 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:15.874413 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.884394 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.955029 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:15.991823 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:15.996802 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.000613 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240503 14:14:16.001623 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:16.005568 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.005998 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.014729 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.027124 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.036248 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.057912 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:14:16.069881 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.074383 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 14:14:16.076726 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.122155 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.125321 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 14:14:16.141196 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:14:16.164333 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.167088 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:16.196251 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.198639 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.211494 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 102 similar messages]
W20240503 14:14:16.228258 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.232380 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.236863 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.284368 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:16.314707 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.344673 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.359683 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:16.364944 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:16.371193 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.372644 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.425813 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.480237 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.497817 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.511258 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.521430 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:16.528797 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.544390 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.697724 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.734674 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:16.746183 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.818316 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:16.823621 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.857043 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:16.890587 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.937474 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:16.944936 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:16.971199 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:16.997656 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.019351 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.072297 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:17.100880 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:17.136816 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:17.168917 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:17.185021 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.240460 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:17.255625 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.286356 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:17.311434 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:17.410528 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:17.469022 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:17.486403 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.518235 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.520609 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.524194 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.595427 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:17.603701 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.606474 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.620851 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:17.646178 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:17.767884 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:17.882236 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:17.893718 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:17.909775 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:14:17.937237 17227 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:17.937445 17227 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:14:17.938331 17227 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:17.939992 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:17.957448 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:17.957811 9469 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:17.959671 17227 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:17.992527 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.025077 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:18.054169 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:18.065842 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.094504 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 603002fdf6b84d3ea63c65636eb42cb1: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:18.143898 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.152057 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.170295 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:18.191964 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.228789 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.247499 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.261874 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.266467 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.271216 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 14:14:18.278936 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.329480 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.344447 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.345695 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:18.351927 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.352955 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.377408 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.428308 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.428464 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.428684 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.471262 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.492659 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.532562 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.543690 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.547505 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.558650 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:18.580603 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.584473 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.586503 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:18.589011 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.626902 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.650095 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.651746 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.683977 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.703955 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.710346 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.765048 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:18.778074 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.810690 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:18.829324 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.860570 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:18.864812 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.935978 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:18.971202 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.016754 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.045568 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.051122 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:19.073072 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:19.085915 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.117471 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.166718 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.176578 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.191845 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.241438 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:19.255704 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.269930 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.290638 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:19.303663 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.376148 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.382563 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.451670 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240503 14:14:19.465337 17811 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:19.465579 17811 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:14:19.466307 17811 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:19.467751 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:19.468986 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:19.469070 9735 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:19.469331 17811 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:19.495275 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:19.549984 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:19.559983 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:19.588652 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.589745 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.616506 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.651494 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:19.695396 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.777942 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.824939 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:19.830077 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:19.839218 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:19.858073 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:19.958178 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:19.958940 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:19.990012 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:20.006481 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.018702 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.104665 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.127424 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.133244 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.159018 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:20.222296 17415 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:20.222492 17415 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:20.223309 17415 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:20.224952 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:20.225395 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:20.225505 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:20.225911 17800 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:20.249485 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.276013 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.363402 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:20.373138 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.443038 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:20.494341 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.503927 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.543197 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:20.553496 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:20.554719 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.556421 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.558461 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.566778 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.694904 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.701646 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.746059 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.766010 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.821909 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.826864 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.827642 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.841512 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.895973 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:20.924345 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.944301 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:20.959231 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 14:14:20.967726 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:20.969532 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:20.995095 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.006713 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.059644 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 14:14:21.060580 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.067119 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.074649 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.091486 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:14:21.100680 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:14:21.101662 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.101648 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.119618 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.124861 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:21.135270 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.155352 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 14:14:21.158890 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:21.173928 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.193863 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.196570 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.210176 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.250324 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.258531 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:14:21.266479 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.303670 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.329836 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.334563 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.351378 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 103 similar messages]
W20240503 14:14:21.380525 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.394793 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:21.511708 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.542740 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.556710 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:21.565984 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:21.568195 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.574546 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.602245 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:21.649621 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.717291 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.719107 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.726162 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.782786 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.798122 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.802784 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.814272 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:21.829180 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.849527 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:21.868045 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.919041 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.921844 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:21.929502 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.951725 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:21.999034 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.009336 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:22.066159 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.066289 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:22.099368 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:22.147310 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:22.175063 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:22.253219 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:22.306342 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:22.419420 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.434540 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:22.452332 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.517735 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.594033 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:22.599153 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.604449 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:22.645205 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:22.647287 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.708794 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:22.720038 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.725736 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:22.757030 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:22.762559 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.822341 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:22.830443 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:22.854847 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:22.914418 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:22.975333 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.082198 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:23.111202 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.169265 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:23.171728 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.194203 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.197170 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.197564 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:23.322922 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.324206 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.327347 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.344394 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:23.365700 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.373227 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 14:14:23.390203 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.410776 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.432606 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.481472 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:23.483815 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.503430 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.523098 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.565002 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.572969 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:23.581379 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.584487 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.626116 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.633898 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.634893 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.653071 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.666473 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.678385 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.718273 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.747452 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:23.776083 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:23.778913 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.791548 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.797612 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.804498 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.820569 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.862401 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.957170 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:23.960310 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:23.983446 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:23.997539 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.000080 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:24.025417 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:24.058637 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.116219 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.134094 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:24.156713 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:24.163372 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:24.196238 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.209873 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:24.222373 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.233889 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:24.240438 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.252527 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.327679 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:24.351202 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.362221 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:24.376248 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.458779 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.533622 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:24.540987 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:24.562674 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.603179 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.604867 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.610352 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:24.626185 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:24.627483 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:24.634562 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.636242 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:24.667575 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:24.701138 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:24.765166 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:24.840970 17811 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:24.841158 17811 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:14:24.841774 17811 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:24.843864 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:24.844956 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:24.845072 9735 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:24.845304 17811 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:24.906132 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.043107 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:25.090642 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:25.096694 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.109040 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.109503 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.130692 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.131618 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.215543 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.244515 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:25.261648 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.269227 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.300817 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.302630 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.358673 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:25.399909 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:25.473402 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.534242 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:25.618597 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:25.653275 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.746634 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 14:14:25.779439 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.794256 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:25.795377 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.816071 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.819679 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.831908 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.835389 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:25.860556 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.874439 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.875018 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.898809 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.904418 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:25.911424 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.965626 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:25.981324 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.019680 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.022696 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.054133 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:26.065871 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:14:26.067411 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.071681 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.099785 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240503 14:14:26.112934 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:14:26.122306 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.160392 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:26.166311 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.187961 9467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 14:14:26.202284 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.252379 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.261842 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.288960 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:26.298895 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.313030 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.328516 6867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:14:26.331612 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.350252 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.358389 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 14:14:26.376374 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.377938 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.387085 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.412134 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.440203 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:26.511145 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.517226 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.524273 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.546104 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:26.562258 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.587922 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.624241 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:26.646556 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.662554 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.677424 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.719259 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.748638 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.778301 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.805222 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.816893 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:26.881888 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:26.911010 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:26.954456 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:27.013131 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.013871 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.016108 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.022112 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.022814 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:27.067795 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.130095 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:27.167495 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:27.224045 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.227859 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.248544 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.258590 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:27.270443 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:27.284031 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.338011 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.341171 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:27.344666 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:27.421284 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.549078 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.550174 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:27.558254 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:27.600926 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.606626 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.632004 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.760113 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.776188 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.801968 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:27.802325 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240503 14:14:27.817232 19124 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:27.817436 19124 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:27.818051 19124 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:27.819850 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.820468 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.835247 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:27.835487 7125 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:27.835997 19124 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:27.898276 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.925748 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:27.927699 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.941462 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:27.959390 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:27.968597 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:28.149467 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 14:14:28.168692 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.260254 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.260254 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.325732 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.334699 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.343688 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.346302 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.385655 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.438156 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.471532 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.477773 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:28.513485 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.522454 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.526628 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:28.536734 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.543190 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:28.557690 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:28.577159 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.602324 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.650398 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.665412 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.677330 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.684218 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.689947 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.714592 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.759070 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.759486 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.761087 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.775476 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:28.857015 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.876616 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.897262 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.904492 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.938792 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:28.945196 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:28.946009 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.000404 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.097530 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:29.109290 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:29.109794 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:29.122438 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.137662 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.150367 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.150851 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.181679 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.208338 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.222121 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.224128 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.292392 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.328889 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.338768 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.372974 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.472760 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.508939 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.513270 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:29.518128 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.527316 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.530053 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:29.543401 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.592342 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:29.599413 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:29.629487 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.658282 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:29.658468 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.710152 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:29.731560 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:29.895689 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.913295 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.917270 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:29.980763 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:29.991819 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.054085 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:30.065696 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:30.069201 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:30.140223 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.149367 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:30.173194 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.192121 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.210958 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.263690 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.337352 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.358978 6644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 14:14:30.360149 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3f2bd341493482f9519e6dd74cd2f3d: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:30.379921 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:14:30.451331 19337 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:30.451546 19337 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:14:30.452383 19337 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:30.453897 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:30.454051 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:30.454180 9469 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:30.454419 19337 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:30.460983 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.469080 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.476599 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:30.483011 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.520465 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:30.526544 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:30.584861 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.623860 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:30.722101 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.761240 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:30.800622 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 14:14:30.800732 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.830148 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.859853 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.910023 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.916272 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.928088 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:30.933583 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:30.957015 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.964963 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:30.983217 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.029255 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.040647 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.063235 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.074363 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240503 14:14:31.104305 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 14:14:31.115024 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.120460 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:14:31.121349 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:31.121553 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.141299 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.166842 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.188614 9467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 14:14:31.190140 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.196403 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.208391 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:31.208516 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.210546 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:31.217610 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240503 14:14:31.218030 19296 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:31.218211 19296 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:31.218839 19296 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:31.220060 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:31.221341 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:31.221455 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:31.221828 19296 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:31.307077 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.310843 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:31.321435 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.330364 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.359129 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.388407 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 96 similar messages]
W20240503 14:14:31.391163 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:14:31.392072 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.396895 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.407840 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.412454 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.415078 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.513890 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.526515 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.531004 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.574862 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.590991 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:31.602856 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.624087 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.724954 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:31.729396 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:31.758008 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.762655 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:31.780191 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.837631 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:31.839002 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.919566 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:31.965884 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.029199 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.032151 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:32.055756 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.078619 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.079708 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.092752 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.100569 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:32.116767 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:32.138739 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:32.143252 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:32.144942 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:32.154945 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.157449 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 14:14:32.229367 19296 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:32.229554 19296 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:32.230157 19296 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:32.231786 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:32.231915 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:32.232031 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:32.232234 19296 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:32.240593 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:32.273718 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.283241 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:32.344576 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:32.414314 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:32.462790 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:32.520978 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:32.524096 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:32.602864 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.602851 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:32.619200 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.626673 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:32.662521 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:32.689788 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:32.746897 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.788125 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:32.804355 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.843135 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.856154 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:32.869163 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:32.930464 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:32.989511 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:32.992969 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.028631 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.055054 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.055795 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:33.089076 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.128355 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.128739 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 14:14:33.148433 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.223969 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.225611 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.236071 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:33.239809 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.277056 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.332834 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.361567 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.507580 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.514464 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:33.517299 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:33.521068 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.530831 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.560555 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.564761 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.571492 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.579555 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:33.597452 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.662914 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240503 14:14:33.664829 19919 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:33.664980 19919 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:14:33.665725 19919 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:33.667434 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:33.667925 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:33.668071 9735 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:33.668287 19919 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:33.710089 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.722543 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.732527 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.768406 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.772310 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.775104 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.775434 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.784186 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.813163 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.817147 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:33.836973 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:33.840981 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.872073 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:33.994720 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:34.001461 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.009846 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.017302 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.036723 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.048300 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.098548 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:34.111393 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:34.151449 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.260385 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.261323 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.328148 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.333386 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:34.376199 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.378858 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.437122 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.456946 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.479904 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.486855 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.497934 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.502846 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.504009 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:34.517580 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 603002fdf6b84d3ea63c65636eb42cb1: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:34.522347 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:34.526536 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:34.526664 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:34.552687 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.565363 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.582384 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:34.584600 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:34.657579 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:34.761436 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:34.845305 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.870358 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:34.912911 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:34.915148 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:34.915576 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:34.981650 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.034886 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:35.053278 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:35.097555 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.121367 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:35.137444 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.147908 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.273613 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.317097 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:35.366143 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:35.380331 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:35.405061 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:35.424295 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:35.433069 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 14:14:35.439371 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.444057 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:35.451404 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.465101 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.519016 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:35.519418 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.542652 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.570096 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:35.583298 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.652604 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:35.653085 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:35.657032 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.715185 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.718257 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.834904 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:35.887966 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:35.888201 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:35.891068 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:35.933694 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:35.936983 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:35.937069 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.033102 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.052882 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.056115 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.071007 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:36.077441 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.101740 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.103725 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 14:14:36.104807 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.129722 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:14:36.132906 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:14:36.138192 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:36.144132 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.157953 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.211274 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.217836 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.235199 9467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240503 14:14:36.236200 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.238034 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.279223 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.320319 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:36.332907 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.359966 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.378172 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:36.391845 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:14:36.392841 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.394862 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.402987 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:14:36.468464 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.481330 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.525245 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.540233 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.574309 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.594512 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:36.670871 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.682010 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.682804 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.721573 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.731696 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:36.756067 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:36.790289 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.873546 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.874450 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.905517 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:36.914726 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:36.960067 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:36.982486 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:36.995376 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:37.018028 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:37.030122 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.059193 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.096825 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:37.097565 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:37.118911 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.155838 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:37.237800 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:37.255764 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:37.256243 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:37.265558 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:37.281550 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:37.338032 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.419751 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.438520 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:37.502959 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:37.534338 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.540195 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.573187 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.623481 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:37.655304 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:37.690069 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.698285 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:37.702493 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:37.823099 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.848498 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:37.905953 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.941401 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:37.947638 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.012490 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:38.016019 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.040784 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.051203 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.059382 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.072861 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:38.083706 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.150251 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:38.154748 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 14:14:38.160246 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.179988 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:38.185567 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.187774 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.282451 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.292534 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.344662 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.349785 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:38.408833 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:38.428453 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.438396 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.439054 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.495046 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.536412 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.550058 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.578583 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.584496 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.584849 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.587216 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:38.674273 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.682379 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.734730 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.756381 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.787005 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.796361 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:38.878643 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:38.890403 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.897504 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.917352 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.919193 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.952894 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:38.968088 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.968070 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.977974 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.978163 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.982877 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:38.988181 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:38.992460 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:39.000560 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:39.007537 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.024776 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:39.052598 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.161919 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.164052 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:39.194120 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:39.300874 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:39.329245 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.365386 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:39.435375 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:39.469064 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:39.485884 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.494380 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.509344 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.537181 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:39.558048 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.565194 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:39.632189 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.650954 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:39.663725 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:39.677667 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:39.710127 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:39.722632 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.753878 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:39.787344 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.788120 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:39.830299 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:39.912523 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:39.941216 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:39.946075 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:40.063864 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.085037 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:40.091120 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:40.094007 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.159955 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:40.188484 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.223245 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:40.230041 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.260231 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.265038 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:40.341449 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:40.379834 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:40.383937 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.485965 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:40.490181 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:40.552623 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.553197 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:40.678460 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.698451 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 14:14:40.707525 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.723487 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.753916 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.760072 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:40.777401 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.808858 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:40.824676 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.828974 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:40.845903 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:40.868101 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.896333 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.922555 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:40.930733 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:40.934834 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:40.935066 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:40.963523 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:40.976119 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:41.017861 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.032168 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.104409 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.148042 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 14:14:41.148873 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.151530 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:14:41.152477 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.159909 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:14:41.160651 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.164835 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.176717 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.178890 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.229950 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.234881 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.238283 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.243626 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.276626 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240503 14:14:41.300696 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.352952 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.356161 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.373996 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.381685 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:41.394115 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:41.406031 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 14:14:41.430750 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:41.432792 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240503 14:14:41.439072 20718 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:41.439280 20718 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
W20240503 14:14:41.439675 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:14:41.440435 20718 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:41.441680 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:41.442116 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:41.442224 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:41.442653 20507 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:41.444792 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:41.451678 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.477324 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.525916 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.566886 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.599056 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240503 14:14:41.609099 20964 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:41.609295 20964 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:14:41.609944 20964 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:41.611384 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:41.611639 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:41.611761 9467 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:41.612710 20964 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:41.633193 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.649514 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.853713 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.854748 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:41.859007 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:41.862701 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:41.873070 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:41.877494 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:41.894651 6648 maintenance_manager.cc:419] P 371d064947984511bcd526e0de3fb910: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033384
W20240503 14:14:41.918779 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 14:14:41.951354 6647 maintenance_manager.cc:643] P 371d064947984511bcd526e0de3fb910: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.056s user 0.044s sys 0.008s Metrics: {"bytes_written":86409,"cfile_init":1,"dirs.queue_time_us":374,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1166,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1438,"lbm_writes_lt_1ms":35,"peak_mem_usage":0,"rows_written":237,"thread_start_us":628,"threads_started":2,"wal-append.queue_time_us":298}
I20240503 14:14:41.952724 6648 maintenance_manager.cc:419] P 371d064947984511bcd526e0de3fb910: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 37233 bytes on disk
I20240503 14:14:41.953580 6647 maintenance_manager.cc:643] P 371d064947984511bcd526e0de3fb910: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 14:14:41.966930 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:41.997501 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.011361 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:42.021515 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:42.023301 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:42.082041 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.104900 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.110687 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:42.121028 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:42.142122 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:42.274709 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.346870 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:42.347730 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.358976 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.365213 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:42.368779 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.430437 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:42.445976 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:42.453059 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:42.489168 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:42.498513 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:42.510950 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:42.520455 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:42.575994 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.593107 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:42.611863 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.641624 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.656906 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.686879 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:42.733438 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:42.804409 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:42.831461 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:42.889708 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:42.944993 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.008195 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:43.053442 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.058234 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.081924 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.190944 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.194402 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.223132 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:43.225495 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 14:14:43.249071 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:43.277216 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.296478 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.311182 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.323050 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.327666 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.344446 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:43.388491 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.404721 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.410421 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.456696 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.459111 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.507787 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.511008 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:43.549407 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.589152 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.596218 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.616336 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.623306 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.640133 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:43.648859 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.658634 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.663348 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.675024 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.761062 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.765527 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.771507 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:43.777621 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:43.790689 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.803068 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.803324 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.813045 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.839468 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.840636 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:43.902158 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.934377 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.938396 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:43.973767 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.037367 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.070327 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:44.155577 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:44.191852 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:44.213101 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.218472 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:44.255582 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:44.258978 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:44.333036 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.383157 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:44.388417 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:44.449905 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:44.475692 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.476006 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:44.544658 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:44.569188 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.596889 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:44.604143 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:44.637723 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.660030 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:44.679414 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:44.692832 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:44.714838 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:44.775645 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.785938 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.801294 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:44.845774 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.878329 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:44.912127 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:44.932832 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:44.942287 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:44.991737 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:45.066561 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:45.084089 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.098975 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:45.103576 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:45.135720 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.181550 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.204054 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.213716 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.260074 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:45.271562 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:45.340409 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.345835 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:45.460871 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.524716 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.546468 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.573331 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:45.576370 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.612685 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:45.623519 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 14:14:45.637689 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:45.645118 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.658974 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:45.674821 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.692387 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:45.692582 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.721957 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:45.777149 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.804518 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:45.815907 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.828737 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.916810 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:45.919956 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:45.941499 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:45.979805 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:45.990947 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:46.038081 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.043375 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.054893 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.111053 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.154835 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.167114 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 14:14:46.168771 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:46.174048 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.181486 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 14:14:46.182415 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.203972 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:46.212529 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 14:14:46.213399 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.221134 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:46.264001 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.279902 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 14:14:46.281002 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.296897 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.307989 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.308799 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.357579 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.358804 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.359880 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.376766 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.386055 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.415556 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:14:46.423472 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.425611 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:46.461341 6867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 14:14:46.462841 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.497684 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.518748 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:46.554980 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.563076 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:46.566958 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.604800 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:46.626435 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:46.756390 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:46.859535 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.902554 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.913380 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:46.929275 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:46.960629 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:46.963366 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:47.000754 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:47.022173 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.031260 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:47.045454 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:47.099931 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.127393 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:47.192255 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:47.202143 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:47.274494 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:47.312727 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:47.319258 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:47.342671 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:47.355230 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.355352 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.413803 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.421940 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.464612 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.490782 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:47.504026 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:47.546519 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:47.598824 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.626654 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.631060 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.643299 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:47.654899 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:47.673336 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.707675 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:47.719357 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.744556 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:47.778808 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.837364 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.948954 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:47.954030 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.010807 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.050127 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 14:14:48.050966 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:48.058251 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:48.148722 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.185525 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:48.204228 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.219511 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.251276 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.255563 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:48.271337 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.295439 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.364504 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.369243 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.371718 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.380501 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.387490 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.414712 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.475441 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.482676 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.502784 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.511883 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:48.548362 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.606632 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.607702 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.625089 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.630131 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:48.662767 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.662793 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.670799 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:48.689157 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.692072 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.696220 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:48.775887 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.802400 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.816212 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.846138 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.861315 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.900645 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.933460 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:48.944624 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:48.962467 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.012225 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.018146 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:49.042438 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:49.050706 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:49.074672 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.133383 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.163724 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.165860 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.176790 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:49.219254 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:49.252166 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:49.298408 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:49.349934 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.408092 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.466820 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.503240 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.566715 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:49.597340 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:49.637369 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:49.640662 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:49.656098 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:49.756026 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:49.761698 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.774416 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.785333 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:49.833292 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:49.833923 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.839952 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.855661 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.909297 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:49.939273 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.941582 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:49.978596 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:49.984771 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.012423 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.056175 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:50.088618 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.097121 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:50.099804 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:50.110931 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.116997 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.155334 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.165582 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.197477 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:50.199419 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.225539 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.277186 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:50.311980 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.465310 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:50.479738 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.483529 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.531301 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:14:50.559737 21915 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:50.559995 21915 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:14:50.560809 21915 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:14:50.562321 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:50.562716 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:50.562820 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:50.563431 21915 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:50.675578 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.684397 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:50.695225 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.715433 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:50.758919 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:50.781512 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.797938 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:50.865293 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:50.904579 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:50.908020 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.938169 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:50.957314 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:50.961058 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 14:14:50.971493 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:50.978560 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.008113 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:51.020771 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.033154 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.060133 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.062557 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.074728 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.075474 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.117484 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.134414 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.150099 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.165513 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.167464 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 14:14:51.171396 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.185478 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:51.203513 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.214998 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:14:51.216301 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.220680 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:14:51.221959 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:51.227332 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:51.261015 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.284933 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 14:14:51.298969 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.340435 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.359845 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.370673 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.398300 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.434247 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:14:51.475195 6867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 14:14:51.549263 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.608151 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:51.618613 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:51.634527 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:51.639802 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:51.675163 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.714982 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:51.717731 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.748757 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:51.756707 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.766201 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.841712 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:51.842535 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:51.895290 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.912475 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:51.960927 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:51.964797 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:14:51.975175 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(dcb23bc5b0a44da7989aa828826ece07): perf score=0.033341
I20240503 14:14:52.009292 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(f56e8ea68eb94475838da245b5f060cc): perf score=0.033349
W20240503 14:14:52.073114 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:52.080461 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:52.118568 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:52.120033 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:52.126206 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:52.133960 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(dcb23bc5b0a44da7989aa828826ece07) complete. Timing: real 0.158s user 0.071s sys 0.040s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":468,"dirs.queue_time_us":19238,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":1050,"drs_written":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":69200,"threads_started":3,"wal-append.queue_time_us":48836}
I20240503 14:14:52.137853 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(dcb23bc5b0a44da7989aa828826ece07): 4838 bytes on disk
I20240503 14:14:52.139104 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(dcb23bc5b0a44da7989aa828826ece07) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 14:14:52.145398 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(f56e8ea68eb94475838da245b5f060cc) complete. Timing: real 0.135s user 0.074s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":376,"dirs.queue_time_us":21473,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1461,"drs_written":1,"lbm_read_time_us":133,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1016,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":26530,"threads_started":3,"wal-append.queue_time_us":4745}
I20240503 14:14:52.150414 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(f56e8ea68eb94475838da245b5f060cc): 4923 bytes on disk
I20240503 14:14:52.157560 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(f56e8ea68eb94475838da245b5f060cc): perf score=0.033389
I20240503 14:14:52.158038 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(f56e8ea68eb94475838da245b5f060cc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240503 14:14:52.182636 20423 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:52.182837 20423 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:14:52.183542 20423 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:52.189914 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:52.190161 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:52.190255 9735 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:52.190770 20423 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:14:52.206275 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:52.248498 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.250452 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:52.251840 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:52.266850 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:52.293994 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:52.294296 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.358877 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.360946 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:52.375448 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.386330 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.398095 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.408962 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 14:14:52.443223 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(f56e8ea68eb94475838da245b5f060cc) complete. Timing: real 0.285s user 0.073s sys 0.022s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":9677,"dirs.queue_time_us":15311,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":1140,"drs_written":1,"lbm_read_time_us":233,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7601,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":38753,"threads_started":3,"wal-append.queue_time_us":11073}
I20240503 14:14:52.447118 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(f56e8ea68eb94475838da245b5f060cc): 4923 bytes on disk
I20240503 14:14:52.448283 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(f56e8ea68eb94475838da245b5f060cc) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
W20240503 14:14:52.455936 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.490473 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.492278 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.572633 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:52.613566 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.619704 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:52.624339 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.624594 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:52.654533 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:52.672502 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:52.689311 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.706254 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.747664 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:52.754750 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.009816 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:53.017318 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.040271 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.075521 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:53.089720 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:53.147367 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.153149 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:53.196895 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.241742 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.249670 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.284793 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.296767 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.338785 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.361275 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.370199 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.402805 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.410701 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.446537 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.455709 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:53.456404 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.456794 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.457803 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.479633 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.488637 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.523262 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:53.528581 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.562618 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:53.586004 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 14:14:53.645434 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.650949 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.672433 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:53.683840 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.685428 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.708173 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.710696 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.727998 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.780045 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.791649 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.798164 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.847505 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.855021 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.860046 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.880895 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:53.885015 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:53.903409 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:54.009677 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:54.101280 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:54.101502 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.117851 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:54.148234 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:54.165218 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:54.273373 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.324466 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.333527 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:54.353832 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:54.362546 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.362591 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:14:54.412923 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(6e0bb82edac2464b9d618c1c80807fb2): perf score=0.033355
I20240503 14:14:54.432389 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(24090e6169dd44f8b47c5cd772934710): perf score=0.033362
W20240503 14:14:54.434985 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:14:54.446578 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(6e0bb82edac2464b9d618c1c80807fb2) complete. Timing: real 0.033s user 0.021s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":355,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":2631,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":788,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":464,"threads_started":2,"wal-append.queue_time_us":367}
I20240503 14:14:54.448523 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(6e0bb82edac2464b9d618c1c80807fb2): 4583 bytes on disk
I20240503 14:14:54.449254 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(6e0bb82edac2464b9d618c1c80807fb2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20240503 14:14:54.466931 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(ca522e5b44fd44289b4b161f27f71d20): perf score=0.033370
I20240503 14:14:54.487639 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(24090e6169dd44f8b47c5cd772934710) complete. Timing: real 0.055s user 0.030s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":275,"dirs.run_wall_time_us":1894,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":13472,"threads_started":2,"wal-append.queue_time_us":13314}
I20240503 14:14:54.490026 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(24090e6169dd44f8b47c5cd772934710): 4838 bytes on disk
I20240503 14:14:54.490797 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(24090e6169dd44f8b47c5cd772934710) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240503 14:14:54.495699 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:54.502377 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:54.504995 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:54.513849 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:54.515583 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(ca522e5b44fd44289b4b161f27f71d20) complete. Timing: real 0.048s user 0.032s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":368,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1459,"drs_written":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":9600,"thread_start_us":785,"threads_started":2,"wal-append.queue_time_us":512}
I20240503 14:14:54.517360 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(24090e6169dd44f8b47c5cd772934710): perf score=0.033382
I20240503 14:14:54.581012 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(24090e6169dd44f8b47c5cd772934710) complete. Timing: real 0.063s user 0.057s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1398,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":789,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":516,"threads_started":1,"wal-append.queue_time_us":625}
I20240503 14:14:54.584158 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(ca522e5b44fd44289b4b161f27f71d20): 4923 bytes on disk
I20240503 14:14:54.584795 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(ca522e5b44fd44289b4b161f27f71d20) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240503 14:14:54.586354 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 14:14:54.587188 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(24090e6169dd44f8b47c5cd772934710): 4838 bytes on disk
I20240503 14:14:54.587821 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(24090e6169dd44f8b47c5cd772934710) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
W20240503 14:14:54.593423 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:54.612349 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:54.635511 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:54.671828 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:54.725528 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.756350 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.820830 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.822664 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:54.826506 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.835140 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:54.837945 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.902122 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:54.903090 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:54.907732 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.921360 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:54.986814 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.049360 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.058933 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:55.060137 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:55.113060 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.153923 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.155652 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:55.206617 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:55.265332 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.310425 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:55.320472 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:55.361929 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.436769 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.446939 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.459319 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:55.505321 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.524139 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:55.577585 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.613626 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:55.660795 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:55.707418 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.723109 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:55.737459 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.746035 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.828055 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.841107 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:55.841353 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:55.847368 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:55.856341 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.874574 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:55.884564 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.926716 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:55.928550 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.944770 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:55.961849 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 14:14:56.007452 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.040374 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.072299 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.073288 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.102536 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:56.125195 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.139552 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.176122 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.178097 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.195125 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.210716 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 14:14:56.229470 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 14:14:56.230239 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:56.239394 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.250306 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.258965 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 14:14:56.274372 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:56.275290 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:56.287817 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 14:14:56.361503 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.367069 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.404738 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.421408 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.432250 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.435355 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:14:56.454569 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.486505 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 14:14:56.530143 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.541157 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:56.544677 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:56.569204 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.601938 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:56.654668 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.668654 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.670419 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:56.677670 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:56.709220 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.733340 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:56.778256 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:56.787569 22786 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:14:56.787773 22786 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:14:56.788498 22786 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:14:56.789877 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:56.790150 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:56.790261 9469 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:14:56.790709 22786 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:14:56.796878 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.803357 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:56.871887 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:56.972270 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:56.988970 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:57.034992 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.068323 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.078773 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:57.093456 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.101017 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:57.104696 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:57.121390 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:57.199311 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:57.223085 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:57.228842 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.242733 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:57.258821 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:57.345537 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.369081 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.381572 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:57.411559 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.420588 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.459733 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:14:57.473016 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(a90c8e47862945f1a873b17da14bb397): perf score=0.033357
I20240503 14:14:57.516882 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(669d722d3eb1475dba616ed72cb775ac): perf score=0.033365
W20240503 14:14:57.525875 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:14:57.540144 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(a90c8e47862945f1a873b17da14bb397) complete. Timing: real 0.067s user 0.044s sys 0.021s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":348,"dirs.run_cpu_time_us":273,"dirs.run_wall_time_us":2068,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":691,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":5920,"threads_started":2,"wal-append.queue_time_us":5775}
I20240503 14:14:57.541841 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(a90c8e47862945f1a873b17da14bb397): 4838 bytes on disk
I20240503 14:14:57.542410 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(a90c8e47862945f1a873b17da14bb397) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 14:14:57.551393 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:57.553392 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:57.584908 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(669d722d3eb1475dba616ed72cb775ac) complete. Timing: real 0.068s user 0.042s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":343,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1691,"drs_written":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4,"lbm_write_time_us":750,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":484,"threads_started":2,"wal-append.queue_time_us":338}
I20240503 14:14:57.586954 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(c8848a40563c43468ac46f8c604cc578): perf score=0.033381
W20240503 14:14:57.590512 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:57.608422 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:14:57.612790 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(669d722d3eb1475dba616ed72cb775ac): perf score=0.033388
W20240503 14:14:57.635277 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:14:57.638859 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(c8848a40563c43468ac46f8c604cc578) complete. Timing: real 0.052s user 0.040s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1675,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":804,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":2304,"thread_start_us":10239,"threads_started":1,"wal-append.queue_time_us":10333}
I20240503 14:14:57.641072 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(c8848a40563c43468ac46f8c604cc578): 4923 bytes on disk
I20240503 14:14:57.641688 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(c8848a40563c43468ac46f8c604cc578) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 14:14:57.643731 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(669d722d3eb1475dba616ed72cb775ac): 4583 bytes on disk
I20240503 14:14:57.662873 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(669d722d3eb1475dba616ed72cb775ac) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":147,"lbm_reads_lt_1ms":4}
W20240503 14:14:57.694933 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:14:57.704972 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(669d722d3eb1475dba616ed72cb775ac) complete. Timing: real 0.092s user 0.056s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":24555,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":4551,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":852,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":30990,"threads_started":2,"wal-append.queue_time_us":6452}
I20240503 14:14:57.709228 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(669d722d3eb1475dba616ed72cb775ac): 4583 bytes on disk
I20240503 14:14:57.709915 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(669d722d3eb1475dba616ed72cb775ac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240503 14:14:57.727872 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:57.841147 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:57.889149 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:57.918258 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:57.962263 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.044153 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:58.087446 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.111529 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:58.112776 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.120841 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.123162 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:58.123623 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:58.139787 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.153950 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.170661 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.236815 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.254349 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.322499 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.327749 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.327978 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.343791 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.354252 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.383702 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 14:14:58.434981 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.451874 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.493445 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:58.494954 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.501727 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.509581 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:58.516032 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.544730 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.567616 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.574759 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.576010 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:58.601191 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.603467 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.605242 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:58.676216 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.686745 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.768445 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.770938 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.898433 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.913022 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.915627 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:58.922787 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.926133 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:58.935797 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:14:58.981583 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:14:58.986269 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:59.007973 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.017534 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:59.043416 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:59.061070 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.083062 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.089242 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:59.114384 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.153198 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:14:59.157807 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:59.205518 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.240625 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:59.332849 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:14:59.374696 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:59.435825 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:59.439101 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.468685 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:14:59.502244 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.543665 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.600477 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.613773 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:59.615718 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:59.622071 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:14:59.643015 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:14:59.682914 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(c95eaefadbff4511b610ab85cdfeb909): perf score=0.033371
W20240503 14:14:59.720634 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:14:59.726684 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(34773b2ab9194e70b3e49357a2da4d8c): perf score=0.033379
W20240503 14:14:59.734670 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:14:59.735286 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(c95eaefadbff4511b610ab85cdfeb909) complete. Timing: real 0.052s user 0.038s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1471,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":4047,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":811,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1636,"threads_started":2,"wal-append.queue_time_us":360}
I20240503 14:14:59.737908 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(c95eaefadbff4511b610ab85cdfeb909): 4583 bytes on disk
I20240503 14:14:59.739825 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(c95eaefadbff4511b610ab85cdfeb909) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
W20240503 14:14:59.741135 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.745627 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 14:14:59.812052 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(c95eaefadbff4511b610ab85cdfeb909): perf score=0.033399
W20240503 14:14:59.814198 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:14:59.831059 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:14:59.868041 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:14:59.884703 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:14:59.895392 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(34773b2ab9194e70b3e49357a2da4d8c) complete. Timing: real 0.168s user 0.059s sys 0.025s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":8972,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":21949,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":966,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13419,"threads_started":2,"wal-append.queue_time_us":544}
I20240503 14:14:59.897437 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(34773b2ab9194e70b3e49357a2da4d8c): 4923 bytes on disk
I20240503 14:14:59.898949 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(34773b2ab9194e70b3e49357a2da4d8c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":507,"lbm_reads_lt_1ms":4}
I20240503 14:14:59.932914 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(c95eaefadbff4511b610ab85cdfeb909) complete. Timing: real 0.121s user 0.032s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":504,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1966,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2689,"threads_started":2,"wal-append.queue_time_us":2450}
I20240503 14:14:59.935750 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(34773b2ab9194e70b3e49357a2da4d8c): perf score=0.033430
W20240503 14:14:59.976812 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:14:59.997728 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(34773b2ab9194e70b3e49357a2da4d8c) complete. Timing: real 0.061s user 0.054s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1814,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":854,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":309,"threads_started":1,"wal-append.queue_time_us":451}
I20240503 14:14:59.999650 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(34773b2ab9194e70b3e49357a2da4d8c): 4923 bytes on disk
I20240503 14:15:00.000279 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(34773b2ab9194e70b3e49357a2da4d8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
I20240503 14:15:00.003003 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(c95eaefadbff4511b610ab85cdfeb909): 4583 bytes on disk
I20240503 14:15:00.003700 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(c95eaefadbff4511b610ab85cdfeb909) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240503 14:15:00.038919 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:00.041182 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.041662 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.103843 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:00.111006 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.130266 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:00.168772 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:00.191674 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.252413 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.262585 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.268815 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.307418 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:00.396807 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:15:00.401266 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.564332 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.570881 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.612504 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.642460 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.652060 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:00.669318 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:00.675725 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.766414 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:00.783277 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.790669 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:00.822548 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:15:00.849062 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.853691 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.888197 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:00.890749 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:00.929486 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.932842 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:00.947716 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.979110 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 14:15:00.995002 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:00.996783 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.021908 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.035945 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.042724 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.049494 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.049819 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.059691 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.066448 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.108757 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.149888 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:01.155313 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.159796 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.198293 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.221029 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:01.230479 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:15:01.231709 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:01.282225 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.299189 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.313418 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 14:15:01.324573 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 14:15:01.324750 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 14:15:01.326822 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.327710 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:01.329967 22410 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:01.330193 22410 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:01.330758 22410 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:01.332341 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:01.332540 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:01.332638 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:01.332837 22410 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:01.368126 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.383913 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:01.386492 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.392444 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:01.409309 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.425320 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.434088 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.441747 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.478590 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:15:01.561523 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240503 14:15:01.617897 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:01.713927 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.761114 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:01.788666 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.810153 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:01.854019 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.859948 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.885679 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:01.886986 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:01.905995 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.908641 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:01.908623 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 14:15:01.918021 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(b3e2d7664e6c45d69f4f8b20dcf0e4d0): perf score=0.033338
W20240503 14:15:01.937960 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:01.946247 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:15:01.955135 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(b3e2d7664e6c45d69f4f8b20dcf0e4d0) complete. Timing: real 0.037s user 0.018s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":351,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1642,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":788,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":466,"threads_started":2,"wal-append.queue_time_us":331}
I20240503 14:15:01.957454 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(b3e2d7664e6c45d69f4f8b20dcf0e4d0): 4583 bytes on disk
I20240503 14:15:01.958302 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(b3e2d7664e6c45d69f4f8b20dcf0e4d0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
W20240503 14:15:02.007326 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:02.007853 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:15:02.011170 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(1c727d4554464442a6a81a8ad4996c84): perf score=0.033368
I20240503 14:15:02.021589 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(343520037eac42c4ac76fa1cc5e26fc3): perf score=0.033370
I20240503 14:15:02.063874 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(343520037eac42c4ac76fa1cc5e26fc3) complete. Timing: real 0.042s user 0.029s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":370,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1637,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":641,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3789,"threads_started":2,"wal-append.queue_time_us":3628}
I20240503 14:15:02.066128 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(343520037eac42c4ac76fa1cc5e26fc3): 4838 bytes on disk
I20240503 14:15:02.066788 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(343520037eac42c4ac76fa1cc5e26fc3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 14:15:02.067757 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(1c727d4554464442a6a81a8ad4996c84) complete. Timing: real 0.056s user 0.025s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":367,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1886,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":4070,"threads_started":2,"wal-append.queue_time_us":7161}
I20240503 14:15:02.070117 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(343520037eac42c4ac76fa1cc5e26fc3): perf score=0.033377
W20240503 14:15:02.101341 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:02.112692 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:15:02.128863 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(343520037eac42c4ac76fa1cc5e26fc3) complete. Timing: real 0.058s user 0.048s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":71,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1687,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":816,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":227,"threads_started":1,"wal-append.queue_time_us":334}
I20240503 14:15:02.132069 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(1c727d4554464442a6a81a8ad4996c84): 4923 bytes on disk
I20240503 14:15:02.132755 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(1c727d4554464442a6a81a8ad4996c84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 14:15:02.135030 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(343520037eac42c4ac76fa1cc5e26fc3): 4838 bytes on disk
I20240503 14:15:02.135694 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(343520037eac42c4ac76fa1cc5e26fc3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
W20240503 14:15:02.149753 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:02.222576 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.229615 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:02.252411 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.277236 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:02.283635 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:02.302543 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:02.328953 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.347621 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:02.375279 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:02.421003 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.427901 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:02.430402 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.459925 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:02.540752 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.617348 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.623250 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.643172 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.660847 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:02.781476 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.801452 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:02.824771 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.870626 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:02.940209 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:02.949131 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:02.973577 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.004105 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.004797 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:03.074280 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.135241 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.135735 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.147708 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:03.167222 6644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240503 14:15:03.168392 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c3f2bd341493482f9519e6dd74cd2f3d: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:03.205525 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.230857 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.276940 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.278370 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.297835 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.298197 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.334110 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.344015 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.360155 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:03.362696 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:03.369158 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.374979 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:03.392880 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.458711 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:03.461053 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.498056 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.511821 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.560729 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 14:15:03.566088 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.599093 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.606187 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.614080 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:03.641053 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.677456 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.685474 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.706699 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.727249 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.743731 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.744786 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.749126 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.796756 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:03.810256 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.846817 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.856192 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.871176 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:03.882171 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:03.895900 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:03.923543 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:03.933815 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:03.974552 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:04.051362 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:15:04.085048 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(2d5781b2310f4814aaebdcb715b7ddac): perf score=0.033336
W20240503 14:15:04.126711 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 14:15:04.153978 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(847ef240dcc64fb794ffcc06e483dc08): perf score=0.033348
I20240503 14:15:04.164054 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(2d5781b2310f4814aaebdcb715b7ddac) complete. Timing: real 0.079s user 0.044s sys 0.029s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6433,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":14020,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1177,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12080,"threads_started":2,"wal-append.queue_time_us":5604}
I20240503 14:15:04.166860 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(2d5781b2310f4814aaebdcb715b7ddac): 4923 bytes on disk
I20240503 14:15:04.167654 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(2d5781b2310f4814aaebdcb715b7ddac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240503 14:15:04.195456 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(847ef240dcc64fb794ffcc06e483dc08) complete. Timing: real 0.041s user 0.028s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":818,"dirs.run_cpu_time_us":310,"dirs.run_wall_time_us":2142,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":917,"threads_started":2,"wal-append.queue_time_us":294}
I20240503 14:15:04.197614 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(847ef240dcc64fb794ffcc06e483dc08): 4583 bytes on disk
I20240503 14:15:04.198204 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(847ef240dcc64fb794ffcc06e483dc08) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240503 14:15:04.205811 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:04.226938 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(2d5781b2310f4814aaebdcb715b7ddac): perf score=0.033375
W20240503 14:15:04.276600 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:04.279209 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(2d5781b2310f4814aaebdcb715b7ddac) complete. Timing: real 0.052s user 0.049s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":383,"dirs.run_cpu_time_us":267,"dirs.run_wall_time_us":1817,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":767,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":532,"threads_started":2,"wal-append.queue_time_us":359}
I20240503 14:15:04.281040 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(2d5781b2310f4814aaebdcb715b7ddac): 4923 bytes on disk
I20240503 14:15:04.281623 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(2d5781b2310f4814aaebdcb715b7ddac) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 14:15:04.283214 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(34f1ed0cfa8948048b3762919481367c): perf score=0.033386
W20240503 14:15:04.309101 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 14:15:04.319213 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(34f1ed0cfa8948048b3762919481367c) complete. Timing: real 0.036s user 0.030s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":89,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1713,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":756,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":224,"threads_started":1,"wal-append.queue_time_us":321}
W20240503 14:15:04.320798 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:04.321566 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(34f1ed0cfa8948048b3762919481367c): 4838 bytes on disk
I20240503 14:15:04.323758 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(34f1ed0cfa8948048b3762919481367c) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 14:15:04.431311 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.447342 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.472649 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.479314 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:04.483783 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:04.490958 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.528472 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:04.537535 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:04.566931 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:04.569183 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.591696 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:04.639662 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:04.669751 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.670068 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.694702 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:04.734815 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:04.751163 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:04.765275 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.767748 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.771559 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:04.773734 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:04.796823 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:04.936053 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.059458 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:05.091346 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.115621 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.130705 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.142671 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:05.149351 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:05.156072 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:05.222596 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:05.245381 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.269268 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.362280 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.373277 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:05.392760 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.421039 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:05.485643 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet bbbd9f57ea9f457ab0ad4e59bc0f34bf: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:05.493825 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:05.505215 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.535205 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.540735 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.558101 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.561627 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:05.594369 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.615007 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:05.616899 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:05.661167 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.670969 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:05.740559 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.746868 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:05.776451 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:05.801708 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.843214 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.900111 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:05.926548 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:05.993512 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.036989 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.041116 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.045796 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.056511 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:06.065410 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.066642 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.075521 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:06.078722 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.100008 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.110312 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.111759 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.134315 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.155522 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.177965 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:06.185151 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.260393 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 14:15:06.261490 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.271878 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 14:15:06.273136 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:06.347024 9467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 115 similar messages]
I20240503 14:15:06.351651 22410 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:06.351868 22410 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:06.352440 22410 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:06.354139 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:06.357419 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:06.357620 7125 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:06.357945 22410 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:06.360457 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.361377 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.378099 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:06.407820 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 14:15:06.410140 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.413512 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 14:15:06.438197 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:06.511878 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:15:06.517706 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.526865 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.536935 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.554656 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.563776 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 14:15:06.569237 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:06.593199 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(f5738d4265f941519b90b07334d06af9): perf score=0.033334
W20240503 14:15:06.609045 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.628971 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:15:06.635186 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(f5738d4265f941519b90b07334d06af9) complete. Timing: real 0.042s user 0.027s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":364,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1725,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":847,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3512,"threads_started":2,"wal-append.queue_time_us":3343}
I20240503 14:15:06.637351 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(f5738d4265f941519b90b07334d06af9): 4923 bytes on disk
I20240503 14:15:06.637948 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(f5738d4265f941519b90b07334d06af9) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":113,"lbm_reads_lt_1ms":4}
W20240503 14:15:06.688879 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:06.689487 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(a98e955a2faf4fc889ec01bd148dd354): perf score=0.033360
W20240503 14:15:06.721256 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:15:06.722158 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(9027b8bfee0443aab7420df3f7257bcb): perf score=0.033371
W20240503 14:15:06.756080 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:06.772903 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(a98e955a2faf4fc889ec01bd148dd354) complete. Timing: real 0.083s user 0.047s sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":403,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":2521,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1013,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":36673,"threads_started":2,"wal-append.queue_time_us":36514}
I20240503 14:15:06.776247 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(a98e955a2faf4fc889ec01bd148dd354): 4838 bytes on disk
I20240503 14:15:06.776957 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(a98e955a2faf4fc889ec01bd148dd354) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":120,"lbm_reads_lt_1ms":4}
W20240503 14:15:06.813443 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:06.815461 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.821516 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:15:06.839777 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(9027b8bfee0443aab7420df3f7257bcb) complete. Timing: real 0.117s user 0.047s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11230,"dirs.run_cpu_time_us":262,"dirs.run_wall_time_us":1586,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":807,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":29708,"threads_started":2,"wal-append.queue_time_us":6799}
I20240503 14:15:06.842404 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(9027b8bfee0443aab7420df3f7257bcb): 4583 bytes on disk
I20240503 14:15:06.843314 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(9027b8bfee0443aab7420df3f7257bcb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
I20240503 14:15:06.848021 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(a98e955a2faf4fc889ec01bd148dd354): perf score=0.033401
W20240503 14:15:06.875218 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:06.894701 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(a98e955a2faf4fc889ec01bd148dd354) complete. Timing: real 0.046s user 0.029s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":77,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":1711,"drs_written":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4,"lbm_write_time_us":907,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":232,"threads_started":1,"wal-append.queue_time_us":327}
I20240503 14:15:06.897241 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(a98e955a2faf4fc889ec01bd148dd354): 4838 bytes on disk
I20240503 14:15:06.898007 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(a98e955a2faf4fc889ec01bd148dd354) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240503 14:15:06.930258 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:06.936698 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:06.954701 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:06.963964 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:07.032202 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.057703 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.074185 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:07.074291 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:07.075069 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:07.122557 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:07.123816 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.139644 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.184340 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:07.203575 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.222045 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.249253 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.266772 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:07.275342 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:07.304452 6644 catalog_manager.cc:4573] TS f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 603002fdf6b84d3ea63c65636eb42cb1: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:07.324913 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.353150 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:07.406136 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.481715 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:07.485198 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:07.601030 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.605031 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:15:07.637161 23980 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:07.637351 23980 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:15:07.637998 23980 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:07.639807 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:07.642570 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:07.642704 9733 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:07.642946 23980 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:07.644816 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:07.657799 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:07.687510 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.724577 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.729151 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.781673 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.817081 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:07.819926 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:07.861757 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:07.889868 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:07.947072 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:07.965672 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:07.973907 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.006271 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:08.064424 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.135996 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.143160 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.172856 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.181789 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.190670 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.196221 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:08.202191 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.215997 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.218346 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:08.238059 24220 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:08.238308 24220 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:15:08.238909 24220 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:08.240584 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:08.241039 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:08.241147 9469 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:08.241407 24220 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:08.280047 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:08.301119 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:08.437505 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.439939 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.445531 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:08.481333 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:08.490101 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.504954 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.505483 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.524853 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:08.539955 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:08.574177 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:08.594764 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:08.651224 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.668387 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.675277 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:08.688074 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 14:15:08.705765 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.754601 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.758168 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:08.764884 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:08.769026 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:08.802702 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.816221 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.889850 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:08.900905 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:08.902319 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:08.995769 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.012472 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.030225 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:09.042816 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:09.050309 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:09.065848 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:09.079795 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.124738 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.125181 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:09.189564 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:09.217237 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.274266 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 14:15:09.303160 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(09a755b35d694a2b89ca1f58f0fa7976): perf score=0.033376
W20240503 14:15:09.322391 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:15:09.359499 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(09a755b35d694a2b89ca1f58f0fa7976) complete. Timing: real 0.056s user 0.049s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":388,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":2095,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":768,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":565,"threads_started":2,"wal-append.queue_time_us":398}
I20240503 14:15:09.361923 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(20c4e7c22c904daebe97897320f6226e): perf score=0.033386
W20240503 14:15:09.371291 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:15:09.401485 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(20c4e7c22c904daebe97897320f6226e) complete. Timing: real 0.039s user 0.026s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1954,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":770,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":244,"threads_started":1,"wal-append.queue_time_us":361}
I20240503 14:15:09.405499 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(20c4e7c22c904daebe97897320f6226e): 4921 bytes on disk
I20240503 14:15:09.406323 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(20c4e7c22c904daebe97897320f6226e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.410001 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(20c4e7c22c904daebe97897320f6226e): perf score=0.033397
I20240503 14:15:09.411355 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(09a755b35d694a2b89ca1f58f0fa7976): 4581 bytes on disk
I20240503 14:15:09.411955 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(09a755b35d694a2b89ca1f58f0fa7976) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 14:15:09.422307 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(4b4363edcc954769a3ab9c658e198c2d): perf score=0.033403
W20240503 14:15:09.433898 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:15:09.461359 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(20c4e7c22c904daebe97897320f6226e) complete. Timing: real 0.051s user 0.033s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4452,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":5332,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1157,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":4929,"threads_started":2,"wal-append.queue_time_us":453}
I20240503 14:15:09.463570 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(20c4e7c22c904daebe97897320f6226e): 4921 bytes on disk
I20240503 14:15:09.464417 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(20c4e7c22c904daebe97897320f6226e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4}
W20240503 14:15:09.470319 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:09.478876 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:15:09.494706 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(4b4363edcc954769a3ab9c658e198c2d) complete. Timing: real 0.072s user 0.034s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":2317,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1736,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":949,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2456,"threads_started":2,"wal-append.queue_time_us":343}
I20240503 14:15:09.497195 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(4b4363edcc954769a3ab9c658e198c2d): 4838 bytes on disk
I20240503 14:15:09.498370 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(4b4363edcc954769a3ab9c658e198c2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
W20240503 14:15:09.499727 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:09.511564 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:09.548878 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:09.549171 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:09.556741 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:09.594406 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.669519 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.702031 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.707064 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.713531 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:09.728055 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.729279 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.750034 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.813347 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.889932 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:09.909246 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.933238 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:09.937829 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:09.939074 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:09.955299 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:09.980063 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:10.050963 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:10.065573 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.095059 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:10.120123 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:10.174157 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.200181 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.220506 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.378734 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:10.398751 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.400975 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.409653 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:10.426265 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:10.437119 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.437532 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:10.442477 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.506597 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:10.523406 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.554479 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.571672 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.577606 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:10.656466 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.672549 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:10.777134 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.781812 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:10.795411 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.799890 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:10.805505 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:10.822870 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:10.846493 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:10.901260 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.904691 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:10.933272 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:10.962118 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:11.035849 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.036533 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.049582 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:11.055994 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.076844 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.130604 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.148519 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.177115 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.193765 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.209247 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.221776 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:11.225467 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.250423 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 14:15:11.251571 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.265400 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.283874 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 14:15:11.284688 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:11.343086 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.357867 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240503 14:15:11.366986 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.375357 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.382750 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.417165 9863 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:15:11.425017 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:11.442231 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.457857 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:15:11.485662 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:11.533505 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.539296 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.540177 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:15:11.541880 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:11.573161 6867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240503 14:15:11.579233 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.602702 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.675276 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.694020 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:11.720364 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.754245 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:11.848991 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.857519 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:11.868867 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:11.900492 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:11.925172 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:11.942220 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.987879 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:11.997025 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:12.042153 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:12.082104 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.090737 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:12.199000 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.203729 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.251807 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:12.280256 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:12.286300 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:12.303210 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:12.307008 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:12.319437 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.323617 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.347306 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:12.358817 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:12.364820 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.382704 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.384991 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:12.441272 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.522188 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.541013 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:12.591636 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:12.624672 24875 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:12.624864 24875 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:15:12.625667 24875 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:12.628005 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:12.646553 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:12.646926 9467 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:12.647336 24875 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:12.678054 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:15:12.689011 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(f9815af88a784ba7a50a43ca776ecd20): perf score=0.033370
W20240503 14:15:12.690488 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.699946 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:15:12.741742 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(f9815af88a784ba7a50a43ca776ecd20): perf score=0.033379
I20240503 14:15:12.742954 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(f9815af88a784ba7a50a43ca776ecd20) complete. Timing: real 0.054s user 0.044s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":357,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":2030,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":815,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":559,"threads_started":2,"wal-append.queue_time_us":402}
I20240503 14:15:12.746269 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(f9815af88a784ba7a50a43ca776ecd20): 4838 bytes on disk
I20240503 14:15:12.762995 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(f9815af88a784ba7a50a43ca776ecd20) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
W20240503 14:15:12.764791 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:12.777443 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(9b154f780fc742fa9b0efd609c307f95): perf score=0.033391
W20240503 14:15:12.837805 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:12.859833 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(f9815af88a784ba7a50a43ca776ecd20) complete. Timing: real 0.118s user 0.055s sys 0.031s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":11950,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":4341,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1444,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":16575,"threads_started":2,"wal-append.queue_time_us":4801}
I20240503 14:15:12.862310 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(f9815af88a784ba7a50a43ca776ecd20): 4838 bytes on disk
I20240503 14:15:12.863445 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(f9815af88a784ba7a50a43ca776ecd20) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":260,"lbm_reads_lt_1ms":4}
I20240503 14:15:12.880162 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(9b154f780fc742fa9b0efd609c307f95) complete. Timing: real 0.098s user 0.048s sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":365,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":1562,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":884,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":516,"threads_started":2,"wal-append.queue_time_us":337}
I20240503 14:15:12.882177 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(9b154f780fc742fa9b0efd609c307f95): 4583 bytes on disk
I20240503 14:15:12.882716 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(9b154f780fc742fa9b0efd609c307f95) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240503 14:15:12.884452 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(e431272976034209b2c40cb0e460e17a): perf score=0.033412
W20240503 14:15:12.907562 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.928894 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:15:12.934226 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(e431272976034209b2c40cb0e460e17a) complete. Timing: real 0.049s user 0.037s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1521,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1121,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1778,"threads_started":1,"wal-append.queue_time_us":1869}
I20240503 14:15:12.936795 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(e431272976034209b2c40cb0e460e17a): 4921 bytes on disk
I20240503 14:15:12.937592 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(e431272976034209b2c40cb0e460e17a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
W20240503 14:15:12.958093 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:12.960716 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:12.967890 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:12.972664 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.017110 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.020207 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.069571 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.111522 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.129441 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:13.185658 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.204005 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.206473 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:13.210356 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:13.291234 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 14:15:13.316443 23680 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:13.316656 23680 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:13.317276 23680 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:13.318854 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:13.319115 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:13.319208 7125 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:13.319522 24162 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:13.321611 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:13.323985 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:13.358894 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.366524 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.379299 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.381732 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:13.408883 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:13.425246 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.438864 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:13.442085 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:13.442792 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:13.549273 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.562148 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.586998 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 14:15:13.607040 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.624720 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:13.654523 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:13.663813 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.692309 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:13.701556 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.705853 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240503 14:15:13.708532 24162 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:13.708715 24162 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:13.709436 24162 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:13.710723 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:13.711024 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:13.711170 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:13.711428 24162 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:13.737128 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.774586 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.778121 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.797158 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:13.809091 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.809247 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:13.823376 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:13.825914 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:13.870826 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:14.015693 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.020117 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.087538 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.126309 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.199623 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.232326 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:14.253696 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:14.255312 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:14.269551 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:14.270377 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.271082 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:14.288161 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.292886 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.312253 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:14.367722 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:14.431170 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.441691 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:14.482599 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:14.551448 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:14.683055 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:14.714347 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.717573 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:14.752344 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.772049 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:14.786079 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.806041 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:14.809749 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:14.811116 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.845149 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.862282 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.869963 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:14.872298 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.873343 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:14.875929 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:14.887390 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:14.952942 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:15:14.958753 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling FlushMRSOp(b8014635e8294062b1c1985a100307ff): perf score=0.033334
W20240503 14:15:14.979247 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:15.011180 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: FlushMRSOp(b8014635e8294062b1c1985a100307ff) complete. Timing: real 0.052s user 0.032s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":349,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":4339,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":554,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":497,"threads_started":2,"wal-append.queue_time_us":383}
I20240503 14:15:15.013139 7237 maintenance_manager.cc:419] P e1bf5fb7402c46829974e2f74cd88708: Scheduling UndoDeltaBlockGCOp(b8014635e8294062b1c1985a100307ff): 4838 bytes on disk
I20240503 14:15:15.013710 7128 maintenance_manager.cc:643] P e1bf5fb7402c46829974e2f74cd88708: UndoDeltaBlockGCOp(b8014635e8294062b1c1985a100307ff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240503 14:15:15.030754 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:15:15.032487 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(21e68127f4f94413b1be984a87bec6a8): perf score=0.033364
I20240503 14:15:15.067957 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(21e68127f4f94413b1be984a87bec6a8) complete. Timing: real 0.035s user 0.022s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":350,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1443,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1213,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":499,"threads_started":2,"wal-append.queue_time_us":357}
I20240503 14:15:15.069680 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling FlushMRSOp(3160f1a600604f258f76614f13429563): perf score=0.033368
I20240503 14:15:15.133844 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling FlushMRSOp(b8014635e8294062b1c1985a100307ff): perf score=0.033386
W20240503 14:15:15.141116 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240503 14:15:15.142290 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: FlushMRSOp(3160f1a600604f258f76614f13429563) complete. Timing: real 0.072s user 0.063s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1481,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":860,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":308,"threads_started":1,"wal-append.queue_time_us":424}
I20240503 14:15:15.144999 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(21e68127f4f94413b1be984a87bec6a8): 4922 bytes on disk
I20240503 14:15:15.145901 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(21e68127f4f94413b1be984a87bec6a8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20240503 14:15:15.148675 6846 maintenance_manager.cc:419] P f1a669e28f014fe287932acf56bebc13: Scheduling UndoDeltaBlockGCOp(3160f1a600604f258f76614f13429563): 4583 bytes on disk
I20240503 14:15:15.149504 6737 maintenance_manager.cc:643] P f1a669e28f014fe287932acf56bebc13: UndoDeltaBlockGCOp(3160f1a600604f258f76614f13429563) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":132,"lbm_reads_lt_1ms":4}
W20240503 14:15:15.173338 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:15.195621 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: FlushMRSOp(b8014635e8294062b1c1985a100307ff) complete. Timing: real 0.061s user 0.046s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":343,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":2374,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1279,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":4864,"thread_start_us":438,"threads_started":2,"wal-append.queue_time_us":293}
I20240503 14:15:15.197794 6977 maintenance_manager.cc:419] P 7186ed8510424fd597173eaf10c57b12: Scheduling UndoDeltaBlockGCOp(b8014635e8294062b1c1985a100307ff): 4838 bytes on disk
I20240503 14:15:15.199112 6868 maintenance_manager.cc:643] P 7186ed8510424fd597173eaf10c57b12: UndoDeltaBlockGCOp(b8014635e8294062b1c1985a100307ff) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240503 14:15:15.201778 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:15.209661 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:15.222525 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.240715 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:15.242305 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.361886 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.378288 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.420853 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.429479 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:15.466372 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:15.529191 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:15.559937 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:15.569723 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.573880 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.598071 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:15.622943 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.653132 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:15.667670 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.695204 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:15.702672 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.706355 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.779239 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.781328 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:15.784884 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.833381 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:15.839035 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:15.839568 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:15.869177 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.874394 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:15.886065 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:15.922578 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:15.972740 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:15.989259 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 14:15:16.045640 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.106079 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:16.128757 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.151067 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.167853 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.212278 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.220249 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:16.249811 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:16.274197 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.278292 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.298549 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.310263 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.313716 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:16.318408 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:16.342141 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.352114 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240503 14:15:16.359305 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.394529 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240503 14:15:16.444624 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 14:15:16.505026 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:15:16.506948 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:16.514639 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.532822 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.545156 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.563027 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.578001 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.602638 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240503 14:15:16.625538 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:16.668581 6867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 114 similar messages]
W20240503 14:15:16.697949 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.707258 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:16.716264 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:16.721658 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:16.722968 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.741884 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:16.764267 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.810735 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.814772 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:16.984510 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:15:17.003896 25385 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:17.004118 25385 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:15:17.004845 25385 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:17.006752 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:17.006842 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:17.006986 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:17.007086 9469 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:17.007304 25385 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:17.027508 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:17.067190 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:17.069139 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.076433 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:17.082558 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.085100 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:17.104769 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:17.108610 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:17.192328 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.254693 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.263222 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.301291 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:17.424321 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.434178 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:17.509258 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:17.547611 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:17.548959 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:17.561154 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.563254 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.585783 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:17.611166 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.615267 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.619557 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:17.624816 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:17.642055 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:17.705783 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.714246 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:17.735738 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.785352 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:17.848026 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.870061 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.920349 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.940127 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:17.992969 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.000561 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:18.039570 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:18.074532 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.077839 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:18.108206 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:18.117998 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:18.119738 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.131129 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.131278 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:18.144918 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.150947 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.164387 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:18.166940 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.195153 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.196516 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:18.327080 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.331650 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.348026 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:18.381027 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:18.402591 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.427475 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:15:18.431846 24162 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:18.432029 24162 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:18.432651 24162 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:18.433954 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:18.435009 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:18.435101 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:18.435314 24162 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:18.439882 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:18.459395 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.463886 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:18.594579 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.615659 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 14:15:18.617385 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:18.640446 25637 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:18.640652 25637 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:15:18.641709 25637 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:18.643044 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:18.643237 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:18.643362 9469 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:18.643620 25637 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:18.648339 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:18.678774 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:18.714421 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:18.718253 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.768337 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.777504 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:18.816931 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.819181 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.819419 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.866888 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:18.913924 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:18.913887 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:18.961290 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:19.087158 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.129882 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.136617 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:19.151062 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:19.196485 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:19.209647 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.218596 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:19.237726 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:19.256767 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:19.271750 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.291220 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:19.293802 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.302139 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.334774 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:19.384389 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:19.440850 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.442037 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.461835 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.477141 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:19.514613 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:19.532294 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:19.550853 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:19.580914 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:19.589835 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.608172 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.642510 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:19.735028 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:19.837575 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:19.859479 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.927254 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.928656 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:19.964963 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:19.984001 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:20.005568 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:20.011929 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:20.035245 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:20.057487 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.079232 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:20.088621 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.091166 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.133435 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:20.202791 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:20.204119 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.216776 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.258373 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.260767 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.275115 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.287421 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.290735 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:20.313081 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:15:20.335083 24162 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:20.335276 24162 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:20.336011 24162 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:20.337631 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:20.338438 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:20.338572 7127 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:20.338838 24162 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:20.339803 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.380152 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.433643 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.496564 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.502645 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:20.527678 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.532368 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.585371 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:20.601711 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:20.608125 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:20.641222 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.663271 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.668123 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:20.670202 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:20.700923 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:20.737464 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:20.780004 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.807643 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:20.824015 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.847096 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:20.863685 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:20.888934 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:20.936367 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:21.004884 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:21.006698 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:21.017769 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.081542 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:21.108388 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.172149 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.175770 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:21.204088 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:21.204244 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.247944 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 14:15:21.259222 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.302959 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.339726 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.370931 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:15:21.372066 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.376111 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.380666 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.415225 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.430023 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 125 similar messages]
W20240503 14:15:21.470850 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 14:15:21.472095 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.480335 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.562256 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 14:15:21.573066 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:21.604643 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:21.620395 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.625975 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:21.640442 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.661895 6734 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 98 similar messages]
W20240503 14:15:21.679543 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:21.681363 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.695098 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240503 14:15:21.774701 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.819062 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:21.831056 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.847903 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:21.851158 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:21.864370 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.937227 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.946571 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:21.952796 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.015013 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.100350 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.147677 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:22.149830 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:22.168251 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:22.187884 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:22.188354 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:22.196169 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:22.228914 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.253055 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:22.290494 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:22.308717 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:22.328945 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.342741 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:22.360191 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:22.399448 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.400753 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.500617 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:22.513046 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:15:22.544013 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(c95eaefadbff4511b610ab85cdfeb909): perf score=0.033344
W20240503 14:15:22.604101 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.613934 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:22.629328 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:15:22.680995 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(c95eaefadbff4511b610ab85cdfeb909) complete. Timing: real 0.136s user 0.075s sys 0.032s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":409,"dirs.queue_time_us":14962,"dirs.run_cpu_time_us":283,"dirs.run_wall_time_us":3385,"drs_written":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1129,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":20832,"threads_started":3,"wal-append.queue_time_us":6139}
W20240503 14:15:22.684101 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:15:22.686257 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(c95eaefadbff4511b610ab85cdfeb909): 4583 bytes on disk
I20240503 14:15:22.687665 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(c95eaefadbff4511b610ab85cdfeb909) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240503 14:15:22.695852 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.708500 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:15:22.709723 26172 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:22.709937 26172 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:22.710564 26172 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:22.712255 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:22.712468 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:22.712616 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:22.712911 26172 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:22.715476 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:22.727538 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:22.743388 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.771454 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.807498 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.815248 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.833730 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:22.842223 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:22.955157 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:15:22.982532 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(21e68127f4f94413b1be984a87bec6a8): perf score=0.033381
W20240503 14:15:22.991083 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.014542 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:23.037040 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.038141 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.070055 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:23.073442 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.088301 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:23.094002 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:23.096987 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:15:23.099207 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(21e68127f4f94413b1be984a87bec6a8) complete. Timing: real 0.116s user 0.064s sys 0.032s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":3347,"dirs.queue_time_us":6347,"dirs.run_cpu_time_us":289,"dirs.run_wall_time_us":12076,"drs_written":1,"lbm_read_time_us":150,"lbm_reads_lt_1ms":4,"lbm_write_time_us":856,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":16216,"threads_started":3,"wal-append.queue_time_us":6497}
I20240503 14:15:23.102483 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(21e68127f4f94413b1be984a87bec6a8): 4922 bytes on disk
I20240503 14:15:23.104471 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(21e68127f4f94413b1be984a87bec6a8) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4}
W20240503 14:15:23.115923 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.142367 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:15:23.194095 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(24090e6169dd44f8b47c5cd772934710): perf score=0.033364
W20240503 14:15:23.202167 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:23.207513 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:23.229528 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:23.258311 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:23.287182 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.287559 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.288812 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:23.296228 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 14:15:23.305759 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(24090e6169dd44f8b47c5cd772934710) complete. Timing: real 0.111s user 0.060s sys 0.030s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":32357,"compiler_manager_pool.run_cpu_time_us":36,"compiler_manager_pool.run_wall_time_us":35,"dirs.queue_time_us":7220,"dirs.run_cpu_time_us":340,"dirs.run_wall_time_us":1881,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":34063,"threads_started":2,"wal-append.queue_time_us":27123}
I20240503 14:15:23.309741 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(24090e6169dd44f8b47c5cd772934710): 4838 bytes on disk
I20240503 14:15:23.311183 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(24090e6169dd44f8b47c5cd772934710) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":157,"lbm_reads_lt_1ms":4}
I20240503 14:15:23.314955 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(a98e955a2faf4fc889ec01bd148dd354): perf score=0.033350
W20240503 14:15:23.320379 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:23.377910 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(a98e955a2faf4fc889ec01bd148dd354) complete. Timing: real 0.063s user 0.047s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":949,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1963,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4141,"threads_started":1,"wal-append.queue_time_us":4239}
I20240503 14:15:23.380570 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(a98e955a2faf4fc889ec01bd148dd354): 4838 bytes on disk
I20240503 14:15:23.381197 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(a98e955a2faf4fc889ec01bd148dd354) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 14:15:23.462405 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(ca522e5b44fd44289b4b161f27f71d20): perf score=0.033375
W20240503 14:15:23.500313 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 14:15:23.550899 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(ca522e5b44fd44289b4b161f27f71d20) complete. Timing: real 0.088s user 0.058s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":912,"dirs.queue_time_us":2016,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":2419,"drs_written":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1114,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":3530,"threads_started":3,"wal-append.queue_time_us":831}
I20240503 14:15:23.554610 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(ca522e5b44fd44289b4b161f27f71d20): 4923 bytes on disk
I20240503 14:15:23.555682 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(ca522e5b44fd44289b4b161f27f71d20) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
W20240503 14:15:23.564189 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.623440 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:23.659094 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:23.668695 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.674261 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:23.675341 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.678401 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.695266 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.727342 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.754148 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.769292 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:23.779844 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:23.801180 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.802049 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:23.810225 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:23.824723 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.827304 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:23.844817 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 14:15:23.906173 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:23.974992 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:24.035763 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:24.040592 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.065062 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:15:24.115664 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(f9815af88a784ba7a50a43ca776ecd20): perf score=0.033365
W20240503 14:15:24.176528 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:15:24.192026 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(f9815af88a784ba7a50a43ca776ecd20) complete. Timing: real 0.076s user 0.054s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3862,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":1803,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":932,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10507,"threads_started":2,"wal-append.queue_time_us":6322}
I20240503 14:15:24.196290 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(f9815af88a784ba7a50a43ca776ecd20): 4838 bytes on disk
I20240503 14:15:24.198696 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(f9815af88a784ba7a50a43ca776ecd20) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":135,"lbm_reads_lt_1ms":4}
W20240503 14:15:24.203562 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:24.253703 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.257601 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.266006 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:24.291873 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:24.313351 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:24.325887 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.333689 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.396414 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:24.403062 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:24.425138 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.435331 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.454710 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:24.480091 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:24.482934 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.543260 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:24.575111 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:24.677297 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.709055 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:24.741408 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:24.743971 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:24.788364 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:24.798064 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:24.819463 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.821972 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.823812 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:24.827898 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.854858 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:24.877291 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:24.929107 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:25.001353 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.051890 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:25.072901 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.072942 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.101889 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.138347 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.142074 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.180528 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:25.248503 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:25.250000 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:25.321789 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.334295 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.400282 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.414507 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:25.432286 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.470235 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.488855 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.507179 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.527026 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:25.545212 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.600988 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.602437 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:25.610932 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:25.616441 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.649118 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.691936 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:25.736773 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:25.782830 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:25.818782 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:25.832096 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:25.864723 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:25.878964 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.890697 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:25.902895 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.953408 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:25.956035 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:25.976852 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:25.988596 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:26.024183 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.038206 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.085711 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:15:26.088634 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(343520037eac42c4ac76fa1cc5e26fc3): perf score=0.033354
W20240503 14:15:26.132681 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.140954 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.142261 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:15:26.145179 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(343520037eac42c4ac76fa1cc5e26fc3) complete. Timing: real 0.056s user 0.043s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6315,"dirs.run_cpu_time_us":321,"dirs.run_wall_time_us":5358,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":976,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":8036,"threads_started":2,"wal-append.queue_time_us":1910}
I20240503 14:15:26.147769 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(343520037eac42c4ac76fa1cc5e26fc3): 4838 bytes on disk
I20240503 14:15:26.148384 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(343520037eac42c4ac76fa1cc5e26fc3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240503 14:15:26.169636 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.231348 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:26.233668 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.306133 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:26.358260 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:26.363627 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.375674 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 14:15:26.393023 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:26.396996 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:15:26.401592 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(e431272976034209b2c40cb0e460e17a): perf score=0.033335
W20240503 14:15:26.429965 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:26.432780 9467 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 117 similar messages]
I20240503 14:15:26.468129 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(e431272976034209b2c40cb0e460e17a) complete. Timing: real 0.066s user 0.031s sys 0.030s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":172,"dirs.run_cpu_time_us":156,"dirs.run_wall_time_us":2037,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":899,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":210,"threads_started":1,"wal-append.queue_time_us":311}
I20240503 14:15:26.470156 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(e431272976034209b2c40cb0e460e17a): 4921 bytes on disk
I20240503 14:15:26.470738 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(e431272976034209b2c40cb0e460e17a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240503 14:15:26.474220 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.502624 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.514770 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 14:15:26.516235 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.537238 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 14:15:26.563066 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:26.571712 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240503 14:15:26.599926 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.617250 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.639247 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.656993 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:15:26.658733 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(3160f1a600604f258f76614f13429563): perf score=0.033361
W20240503 14:15:26.673857 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:26.690795 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.723672 6867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240503 14:15:26.725893 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 100 similar messages]
I20240503 14:15:26.731245 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(3160f1a600604f258f76614f13429563) complete. Timing: real 0.072s user 0.048s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":470,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":2286,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":890,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":617,"threads_started":2,"wal-append.queue_time_us":363}
I20240503 14:15:26.733685 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(3160f1a600604f258f76614f13429563): 4583 bytes on disk
I20240503 14:15:26.734247 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(3160f1a600604f258f76614f13429563) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240503 14:15:26.765693 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.782562 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.786079 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:26.790153 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.799863 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:26.819052 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.862609 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:26.926666 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:26.946856 26659 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:26.947098 26659 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:26.947840 26659 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:26.949242 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:26.949985 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:26.950116 7125 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:26.950572 26659 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:26.980273 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:26.998950 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.016743 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:27.058133 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.073314 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:27.149433 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 14:15:27.201442 26744 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:27.201653 26744 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:15:27.202302 26744 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:27.203815 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:27.204005 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:27.204108 9735 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:27.204335 26744 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:27.209904 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.211002 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.213410 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:15:27.229414 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(34f1ed0cfa8948048b3762919481367c): perf score=0.033377
W20240503 14:15:27.242002 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:27.265550 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(34f1ed0cfa8948048b3762919481367c) complete. Timing: real 0.036s user 0.014s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":360,"dirs.run_cpu_time_us":264,"dirs.run_wall_time_us":1775,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":737,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":510,"threads_started":2,"wal-append.queue_time_us":357}
I20240503 14:15:27.267557 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(34f1ed0cfa8948048b3762919481367c): 4838 bytes on disk
I20240503 14:15:27.268489 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(34f1ed0cfa8948048b3762919481367c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
W20240503 14:15:27.305227 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:27.309692 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.314312 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:27.326617 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.346050 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:27.359457 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:27.396354 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:27.440308 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:27.451260 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:27.452421 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:27.492223 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:27.501106 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.579432 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.591881 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.594347 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.719630 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:27.729516 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:27.767791 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.808490 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.832186 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.839828 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.875578 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:27.959573 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:27.964455 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.000238 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.012419 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:28.015092 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:28.025097 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:28.039840 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.092268 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:28.107884 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:28.117182 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.134112 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:28.162696 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.176999 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:28.193104 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.223529 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.306864 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.338526 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:28.359629 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.387727 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.389328 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:28.417646 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.442883 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:28.463977 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.466444 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:28.518303 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.610321 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:28.615742 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:28.631562 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:28.636709 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:28.649314 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:28.665051 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.692935 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.696136 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.713089 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:28.739241 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(b3e2d7664e6c45d69f4f8b20dcf0e4d0): perf score=0.033401
W20240503 14:15:28.776672 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:15:28.787559 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(b3e2d7664e6c45d69f4f8b20dcf0e4d0) complete. Timing: real 0.048s user 0.036s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6561,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1453,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1309,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6908,"threads_started":2,"wal-append.queue_time_us":348}
I20240503 14:15:28.789819 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(b3e2d7664e6c45d69f4f8b20dcf0e4d0): 4583 bytes on disk
I20240503 14:15:28.790486 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(b3e2d7664e6c45d69f4f8b20dcf0e4d0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
W20240503 14:15:28.793670 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.793792 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.814621 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:28.855568 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 14:15:28.875511 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:28.880484 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:28.898450 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:28.900398 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:28.948074 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:15:29.005581 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(847ef240dcc64fb794ffcc06e483dc08): perf score=0.033360
W20240503 14:15:29.008638 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:15:29.055482 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(847ef240dcc64fb794ffcc06e483dc08) complete. Timing: real 0.049s user 0.033s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":372,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1572,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":843,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":563,"threads_started":2,"wal-append.queue_time_us":450}
W20240503 14:15:29.056596 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:29.057528 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(847ef240dcc64fb794ffcc06e483dc08): 4583 bytes on disk
I20240503 14:15:29.058554 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(847ef240dcc64fb794ffcc06e483dc08) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
W20240503 14:15:29.059772 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.065936 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:29.109488 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:29.147773 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:29.162583 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:29.167024 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.181097 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:29.299286 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:29.307277 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:29.309216 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:29.318420 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.393656 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.440048 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.463805 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:29.512110 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.533578 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:29.548573 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(9027b8bfee0443aab7420df3f7257bcb): perf score=0.033362
I20240503 14:15:29.564474 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(a90c8e47862945f1a873b17da14bb397): perf score=0.033375
W20240503 14:15:29.635905 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.637634 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:29.648829 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 14:15:29.652875 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(9027b8bfee0443aab7420df3f7257bcb) complete. Timing: real 0.104s user 0.052s sys 0.027s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":6408,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":9198,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1624,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":12803,"threads_started":2,"wal-append.queue_time_us":6358}
I20240503 14:15:29.655529 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(9027b8bfee0443aab7420df3f7257bcb): 4583 bytes on disk
I20240503 14:15:29.656283 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(9027b8bfee0443aab7420df3f7257bcb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240503 14:15:29.686328 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(a90c8e47862945f1a873b17da14bb397) complete. Timing: real 0.121s user 0.074s sys 0.036s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":32033,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":10442,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":989,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":1408,"thread_start_us":38383,"threads_started":2,"wal-append.queue_time_us":6340}
I20240503 14:15:29.688968 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(1c727d4554464442a6a81a8ad4996c84): perf score=0.033334
W20240503 14:15:29.746836 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:29.748327 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:29.751960 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(1c727d4554464442a6a81a8ad4996c84) complete. Timing: real 0.063s user 0.041s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":781,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2067,"threads_started":1,"wal-append.queue_time_us":2505}
I20240503 14:15:29.754420 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(1c727d4554464442a6a81a8ad4996c84): 4923 bytes on disk
I20240503 14:15:29.762954 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(1c727d4554464442a6a81a8ad4996c84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 14:15:29.766302 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(a90c8e47862945f1a873b17da14bb397): 4838 bytes on disk
I20240503 14:15:29.767078 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(a90c8e47862945f1a873b17da14bb397) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240503 14:15:29.778723 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:29.794440 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:29.814918 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:29.828761 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.829635 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.832808 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.838379 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.858867 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:29.909809 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:29.910588 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(1d3ec4f0833d4d489f774757312efb32): perf score=0.033379
W20240503 14:15:29.918751 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:15:29.945819 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(1d3ec4f0833d4d489f774757312efb32) complete. Timing: real 0.035s user 0.025s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1624,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":704,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":269,"threads_started":1,"wal-append.queue_time_us":393}
I20240503 14:15:29.947777 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(1d3ec4f0833d4d489f774757312efb32): 4838 bytes on disk
I20240503 14:15:29.948468 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(1d3ec4f0833d4d489f774757312efb32) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
W20240503 14:15:29.963671 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:30.034471 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:30.094897 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.109439 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:30.115736 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.163946 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.172724 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.182452 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.188874 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:30.197170 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:30.232209 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:30.247833 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:30.249886 26986 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:30.250077 26986 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:15:30.250692 26986 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:30.252317 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:30.252452 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:30.252584 9467 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:30.252795 26986 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:30.300904 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.318936 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.321213 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.343145 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.385404 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.392612 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:30.393805 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:30.423539 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.488766 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:30.542847 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:15:30.547459 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(f56e8ea68eb94475838da245b5f060cc): perf score=0.033363
W20240503 14:15:30.567230 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:15:30.600731 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(f56e8ea68eb94475838da245b5f060cc) complete. Timing: real 0.053s user 0.046s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":355,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1509,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":805,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":471,"threads_started":2,"wal-append.queue_time_us":333}
I20240503 14:15:30.602505 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(f56e8ea68eb94475838da245b5f060cc): 4923 bytes on disk
I20240503 14:15:30.603070 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(f56e8ea68eb94475838da245b5f060cc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 14:15:30.649566 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:30.652382 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.661168 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:30.677186 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:30.757869 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.760838 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:30.766724 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.823150 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.852932 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:30.866037 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.868474 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:30.907037 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:30.928961 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:30.936870 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.979892 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:30.992882 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:30.998935 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.049023 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:31.098193 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:31.115264 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:31.119299 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.147037 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 14:15:31.147318 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:31.184774 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.197494 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.202277 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.239188 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.294032 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.338728 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.355945 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.368016 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.389467 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:15:31.451202 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240503 14:15:31.451726 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.453956 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.467435 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:31.481554 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:31.482894 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.510679 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.519505 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:31.536504 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.576301 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240503 14:15:31.577263 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:31.594918 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.601918 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240503 14:15:31.622967 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:31.629679 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.684967 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:31.737053 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:31.749763 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:15:31.758299 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:15:31.949123 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.957293 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:31.965682 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(09a755b35d694a2b89ca1f58f0fa7976): perf score=0.033350
W20240503 14:15:31.969202 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:31.973019 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:31.973207 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:31.981550 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:32.014907 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.018672 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:32.024045 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:32.029732 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(09a755b35d694a2b89ca1f58f0fa7976) complete. Timing: real 0.064s user 0.040s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":362,"dirs.run_cpu_time_us":240,"dirs.run_wall_time_us":1531,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":855,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6867,"threads_started":2,"wal-append.queue_time_us":6532}
I20240503 14:15:32.032176 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(09a755b35d694a2b89ca1f58f0fa7976): 4581 bytes on disk
I20240503 14:15:32.032747 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(09a755b35d694a2b89ca1f58f0fa7976) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
W20240503 14:15:32.051563 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.074235 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:32.169893 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.198591 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:32.234392 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:32.245568 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:32.273573 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.335247 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:32.367926 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(c8848a40563c43468ac46f8c604cc578): perf score=0.033403
W20240503 14:15:32.378458 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.399726 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:32.405108 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:32.421120 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 14:15:32.422406 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(c8848a40563c43468ac46f8c604cc578) complete. Timing: real 0.054s user 0.033s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":280,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1490,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":869,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":1792,"thread_start_us":434,"threads_started":2,"wal-append.queue_time_us":366}
W20240503 14:15:32.424283 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:32.424887 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(c8848a40563c43468ac46f8c604cc578): 4923 bytes on disk
I20240503 14:15:32.425539 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(c8848a40563c43468ac46f8c604cc578) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 14:15:32.439663 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:32.454177 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:32.459874 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.467782 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:32.476692 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:32.551394 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:32.596271 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:32.597257 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:32.606702 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.612818 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.635960 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.646065 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.663076 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.703706 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:32.703917 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.718225 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.818521 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.866788 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:32.897064 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:32.901108 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:32.967459 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.004920 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:33.023469 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:33.032512 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:33.034763 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:33.079672 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.097021 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.098155 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:33.111127 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.126405 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.132498 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:33.170037 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:33.198534 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.238502 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:33.249830 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.260056 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:33.297542 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(4b4363edcc954769a3ab9c658e198c2d): perf score=0.033397
W20240503 14:15:33.336512 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:33.338591 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(4b4363edcc954769a3ab9c658e198c2d) complete. Timing: real 0.041s user 0.026s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":329,"dirs.run_cpu_time_us":250,"dirs.run_wall_time_us":1581,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":924,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":549,"threads_started":2,"wal-append.queue_time_us":433}
I20240503 14:15:33.340376 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(4b4363edcc954769a3ab9c658e198c2d): 4838 bytes on disk
I20240503 14:15:33.340899 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(4b4363edcc954769a3ab9c658e198c2d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240503 14:15:33.361780 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:33.380077 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.499121 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 14:15:33.547133 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(dcb23bc5b0a44da7989aa828826ece07): perf score=0.033369
W20240503 14:15:33.558936 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:33.564455 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:33.582813 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.599256 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:33.604844 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(dcb23bc5b0a44da7989aa828826ece07) complete. Timing: real 0.057s user 0.041s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":7260,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":2566,"drs_written":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"lbm_write_time_us":940,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7606,"threads_started":2,"wal-append.queue_time_us":313}
I20240503 14:15:33.606961 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(dcb23bc5b0a44da7989aa828826ece07): 4838 bytes on disk
I20240503 14:15:33.611346 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(dcb23bc5b0a44da7989aa828826ece07) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":123,"lbm_reads_lt_1ms":4}
W20240503 14:15:33.639941 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:33.657653 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:33.667655 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.682694 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.692646 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 14:15:33.711675 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:33.717672 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.737425 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.763226 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.785310 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.811144 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:33.839900 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:33.879829 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:33.934520 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:33.936139 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:33.957564 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:34.007369 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:34.054922 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:34.071784 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.079489 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.109791 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:34.140659 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:34.169767 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:34.192767 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:34.193454 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:34.203140 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.315825 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.385365 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.394243 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:34.430151 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.447491 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.454612 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:34.456941 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.604229 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:34.615475 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.667706 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:34.683357 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:34.695755 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.716403 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:34.737175 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.750715 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.792527 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.795300 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.819415 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:34.841270 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:34.894439 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:34.989754 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:34.998417 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:35.006551 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:35.014439 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.015086 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:35.031806 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:35.040463 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:35.059931 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:35.091141 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.106920 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:35.140297 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:35.195003 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.195209 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:15:35.204329 27730 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:35.204527 27730 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:15:35.205220 27730 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:35.206674 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:35.208582 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:35.208712 9733 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:35.208938 27730 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:35.212638 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:35.246052 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:35.256450 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.294764 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.344108 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:35.364147 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.380827 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:35.381767 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.408773 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:35.481864 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:35.542513 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.547240 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.558878 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:35.574153 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.599237 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.626248 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:35.635123 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:35.650099 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.683033 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:15:35.703536 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(34773b2ab9194e70b3e49357a2da4d8c): perf score=0.033381
W20240503 14:15:35.704095 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.726334 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:15:35.765369 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(34773b2ab9194e70b3e49357a2da4d8c) complete. Timing: real 0.061s user 0.041s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6339,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":7676,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":763,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":12889,"threads_started":2,"wal-append.queue_time_us":6302}
I20240503 14:15:35.767948 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(34773b2ab9194e70b3e49357a2da4d8c): 4923 bytes on disk
W20240503 14:15:35.768862 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 14:15:35.768777 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(34773b2ab9194e70b3e49357a2da4d8c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":137,"lbm_reads_lt_1ms":4}
W20240503 14:15:35.802171 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:35.809684 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.849242 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.872738 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:35.905979 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.929396 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:35.936715 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:36.001668 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.053977 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:36.066480 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.150748 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:36.161583 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.205670 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.225309 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:36.234113 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 14:15:36.282760 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.299997 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.304975 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.313479 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.326169 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:36.378391 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.390339 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:36.405524 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.417316 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240503 14:15:36.429464 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.449997 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:36.457317 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.464295 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:36.468201 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:36.482614 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 14:15:36.485097 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:36.503479 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.560226 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.632879 9735 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:15:36.643900 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.684212 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.705158 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240503 14:15:36.712702 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.727315 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:36.730243 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.758306 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240503 14:15:36.768960 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:15:36.807097 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.820226 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:36.852339 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.852751 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.863493 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:36.885322 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:36.891060 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:36.923216 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:36.924099 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.132189 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.212873 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:37.217799 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.241902 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:37.255923 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.264797 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:37.325229 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.339262 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.374591 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.386266 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:37.403630 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:37.406210 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:37.409202 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.421308 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.423205 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.424094 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.436738 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:37.450450 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.504436 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.530969 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.589522 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.606483 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.686705 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.718652 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.765393 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.803547 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:37.818398 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.819069 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:37.838717 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.856155 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:37.857501 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:37.871577 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:37.898394 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:37.954952 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:37.973138 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.012331 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.044054 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.061343 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.065354 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.105032 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.182950 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.195668 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.197062 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:38.234910 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:38.261974 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.278895 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:38.309700 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:38.360822 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:38.391943 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:38.392210 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.416980 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.435326 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.436911 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:38.497291 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:38.550146 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:38.610185 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:38.630548 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.632711 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.634052 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:38.654512 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.672753 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.711122 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:38.715348 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.721283 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:38.746327 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:38.751547 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:38.783481 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.814316 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.827440 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:38.829833 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:38.848052 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:38.891019 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:38.908215 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:38.918974 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 14:15:38.942757 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:38.977636 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:39.033557 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:39.037328 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.059047 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:39.089000 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:39.136561 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(9b154f780fc742fa9b0efd609c307f95): perf score=0.033397
W20240503 14:15:39.178340 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:39.189572 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(9b154f780fc742fa9b0efd609c307f95) complete. Timing: real 0.053s user 0.030s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1349,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1878,"drs_written":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4,"lbm_write_time_us":883,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3446,"threads_started":2,"wal-append.queue_time_us":2318}
I20240503 14:15:39.192327 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(9b154f780fc742fa9b0efd609c307f95): 4583 bytes on disk
I20240503 14:15:39.192970 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(9b154f780fc742fa9b0efd609c307f95) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240503 14:15:39.285272 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:39.315058 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.339870 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.398821 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.434160 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:39.456954 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:39.488785 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:39.514169 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.534726 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:39.543176 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.568912 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.582386 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:39.583643 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.642992 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 14:15:39.650894 28516 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:39.651382 28516 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:39.652273 28516 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:39.653888 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:39.655038 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:39.655155 7125 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:39.655563 27942 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:39.671888 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:15:39.698453 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling FlushMRSOp(a44ed92d65114bfda2e0f38054b17ada): perf score=0.033342
W20240503 14:15:39.741392 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:39.750890 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.752430 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 14:15:39.752398 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: FlushMRSOp(a44ed92d65114bfda2e0f38054b17ada) complete. Timing: real 0.054s user 0.035s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":471,"dirs.run_cpu_time_us":234,"dirs.run_wall_time_us":3093,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":681,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":483,"threads_started":2,"wal-append.queue_time_us":364}
I20240503 14:15:39.754732 9845 maintenance_manager.cc:419] P 3efdbd3506274dba846ae40404f0572c: Scheduling UndoDeltaBlockGCOp(a44ed92d65114bfda2e0f38054b17ada): 4583 bytes on disk
I20240503 14:15:39.755615 9736 maintenance_manager.cc:643] P 3efdbd3506274dba846ae40404f0572c: UndoDeltaBlockGCOp(a44ed92d65114bfda2e0f38054b17ada) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20240503 14:15:39.792737 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.855835 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:39.868432 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:39.912817 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:39.913434 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.946825 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.950557 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:39.955716 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:39.957917 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:39.975415 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.008852 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:40.017297 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.020829 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.023016 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.055244 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.099236 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.177096 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.212047 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.224481 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.233105 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.282546 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.310326 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.362560 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.363920 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:40.383514 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:40.414096 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:15:40.424098 28516 raft_consensus.cc:482] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:40.424350 28516 raft_consensus.cc:504] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:40.425156 28516 leader_election.cc:288] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:40.426950 7127 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:40.427152 7125 leader_election.cc:334] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:40.427237 7125 leader_election.cc:302] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:40.427472 28516 raft_consensus.cc:2726] T 458ebc1295ee49999586eb699a29bbf8 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:40.436803 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.455086 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.459136 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.463624 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.465232 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:40.475776 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.488530 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:40.632411 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:40.669301 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.695026 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.701857 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.738601 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.774484 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.801648 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:40.838690 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.839079 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.844889 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:40.894027 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:40.957298 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:40.960609 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:40.987864 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:40.990157 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.059876 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.079088 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:41.105199 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.127784 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:41.158150 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:41.172046 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:41.178697 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:41.272213 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.275471 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.289386 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.314937 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:41.332502 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.338877 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:41.353170 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:41.378726 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:41.381230 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.394245 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.423044 7125 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 149 similar messages]
W20240503 14:15:41.461902 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.505985 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240503 14:15:41.507925 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.508988 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 14:15:41.511767 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.568231 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.584007 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.613102 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.651835 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 14:15:41.689927 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.696121 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:41.728416 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.755271 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.756511 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 14:15:41.793130 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.809551 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 101 similar messages]
W20240503 14:15:41.839947 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:41.899881 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:41.902276 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:41.910732 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:15:41.911424 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:41.932209 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:41.955318 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:41.999203 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.041208 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:42.070843 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.094398 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.098018 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:42.136997 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:42.169658 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.170533 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.217034 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.225543 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:42.266387 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.286051 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:42.307264 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.326882 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:42.389361 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:42.409624 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.421720 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:42.423146 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.445825 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:42.447027 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:42.449090 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:42.467677 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.513983 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.523984 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.590400 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:42.677656 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.704396 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:42.705492 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:42.725183 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.768730 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.800684 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.808063 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:42.811906 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.821625 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:15:42.826892 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(9027b8bfee0443aab7420df3f7257bcb): perf score=0.033339
W20240503 14:15:42.841883 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:15:42.887701 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(9027b8bfee0443aab7420df3f7257bcb) complete. Timing: real 0.060s user 0.043s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4089,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1909,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":796,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":14473,"threads_started":2,"wal-append.queue_time_us":10592}
I20240503 14:15:42.891304 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(9027b8bfee0443aab7420df3f7257bcb): 4583 bytes on disk
I20240503 14:15:42.892083 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(9027b8bfee0443aab7420df3f7257bcb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
W20240503 14:15:42.898409 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:42.952052 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:42.958437 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:42.972327 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:42.996169 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:43.003473 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:43.029588 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:43.132925 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.136729 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.169009 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:43.180269 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.197556 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:43.209661 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.348693 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.361151 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:43.394867 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:43.403717 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:43.415042 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:43.423411 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:43.440039 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.446081 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:43.485250 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:43.500005 28856 raft_consensus.cc:482] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:43.500221 28856 raft_consensus.cc:504] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } } peers { permanent_uuid: "e1bf5fb7402c46829974e2f74cd88708" member_type: VOTER last_known_addr { host: "127.0.186.4" port: 44337 } } peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } attrs { promote: false } }
I20240503 14:15:43.500988 28856 leader_election.cc:288] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971), 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691)
W20240503 14:15:43.506222 7125 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:43.508073 7127 leader_election.cc:334] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:43.508236 7127 leader_election.cc:302] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1bf5fb7402c46829974e2f74cd88708; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:43.508586 28856 raft_consensus.cc:2726] T 62d262e208a84c14bd296e0c44947813 P e1bf5fb7402c46829974e2f74cd88708 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:43.532753 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:43.567646 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.592281 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:43.615087 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.742384 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.805657 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.819792 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:43.835995 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:43.836987 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:43.850173 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:43.862123 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:43.869491 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:43.871047 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:15:43.912492 28068 raft_consensus.cc:482] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:43.912709 28068 raft_consensus.cc:504] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } } peers { permanent_uuid: "3efdbd3506274dba846ae40404f0572c" member_type: VOTER last_known_addr { host: "127.0.186.7" port: 42783 } attrs { promote: false } }
I20240503 14:15:43.913468 28068 leader_election.cc:288] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:43.915443 9735 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:43.920789 9733 leader_election.cc:334] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:43.920917 9733 leader_election.cc:302] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3efdbd3506274dba846ae40404f0572c; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:43.921134 28068 raft_consensus.cc:2726] T a44ed92d65114bfda2e0f38054b17ada P 3efdbd3506274dba846ae40404f0572c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 14:15:43.930560 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.013134 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:44.013376 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.038288 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.081704 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:44.085981 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.113662 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.126181 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:44.149588 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.174742 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.186712 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.199903 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.211402 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 14:15:44.244798 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.259831 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.263891 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:44.307320 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.351676 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.404268 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:44.489696 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.530210 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:44.533949 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:44.538659 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:44.561663 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:44.590471 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.630569 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.634794 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.657964 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.661126 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.687988 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:44.707880 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.727458 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.735839 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.812211 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.841782 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:44.942886 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:44.968187 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:44.976758 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:44.979957 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:44.987881 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:45.031049 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.037130 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.059000 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.076733 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:45.110245 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.131062 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.139042 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:45.163367 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.210983 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.225157 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.230844 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.238126 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.262785 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:45.276288 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.297789 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.349009 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.362664 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.492170 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:45.494061 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.530107 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.555780 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:45.565377 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.589552 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.629978 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.651988 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:45.669363 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(3160f1a600604f258f76614f13429563): perf score=0.033358
W20240503 14:15:45.686700 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:45.723649 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
I20240503 14:15:45.765811 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(3160f1a600604f258f76614f13429563) complete. Timing: real 0.095s user 0.042s sys 0.026s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":334,"dirs.run_cpu_time_us":221,"dirs.run_wall_time_us":1918,"drs_written":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":17574,"threads_started":2,"wal-append.queue_time_us":3706}
I20240503 14:15:45.769809 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling UndoDeltaBlockGCOp(3160f1a600604f258f76614f13429563): 4583 bytes on disk
I20240503 14:15:45.771848 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: UndoDeltaBlockGCOp(3160f1a600604f258f76614f13429563) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":141,"lbm_reads_lt_1ms":4}
W20240503 14:15:45.773727 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.774914 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.820289 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.854916 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.908509 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:45.912688 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:45.924381 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:45.955919 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 14:15:45.974858 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(6e0bb82edac2464b9d618c1c80807fb2): perf score=0.033399
W20240503 14:15:45.986183 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 14:15:46.009161 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(6e0bb82edac2464b9d618c1c80807fb2) complete. Timing: real 0.034s user 0.030s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":381,"dirs.run_cpu_time_us":336,"dirs.run_wall_time_us":1922,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":854,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":427,"threads_started":2,"wal-append.queue_time_us":295}
I20240503 14:15:46.012167 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(6e0bb82edac2464b9d618c1c80807fb2): 4583 bytes on disk
I20240503 14:15:46.012917 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(6e0bb82edac2464b9d618c1c80807fb2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240503 14:15:46.029631 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:46.049418 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.093317 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:46.111217 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.116184 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.172549 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:46.179819 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.180073 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:46.197918 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:46.325177 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:46.404121 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:46.412515 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.428671 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.438884 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.439224 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:46.465592 7127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240503 14:15:46.465912 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.482069 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.509770 9469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240503 14:15:46.566573 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.582852 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 14:15:46.598085 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:46.627313 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.628667 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.634552 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.641898 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.660367 9733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240503 14:15:46.662842 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.663120 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.667714 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:46.754979 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.756711 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:46.763785 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.787009 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:46.801597 9865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240503 14:15:46.802611 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.818594 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.823881 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:46.844514 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:46.876374 6736 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111) [suppressed 100 similar messages]
W20240503 14:15:46.885784 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.932193 6865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240503 14:15:46.934134 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.960206 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:46.978884 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:46.983465 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.006628 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:47.099292 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.129444 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.183576 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:47.193945 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.198324 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.206867 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:47.240433 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:47.348978 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:47.364398 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:47.377300 6865 consensus_peers.cc:485] T b6d600a8b0134a018f8c9e5a1d9b79c5 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:47.442538 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.452575 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:47.462641 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:47.466552 6865 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:47.487504 6736 consensus_peers.cc:485] T a90c8e47862945f1a873b17da14bb397 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:47.563946 9733 consensus_peers.cc:485] T 412af3d40608416591ed3bf3bc450d67 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.610764 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.653970 6736 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.687572 9467 consensus_peers.cc:485] T 565296ab89c24009835aefefa06fd205 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.707661 6734 consensus_peers.cc:485] T 933c08460b9745feab9bcb56c27bdbb9 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.710136 9865 consensus_peers.cc:485] T a2ed8f0fd81a43f288295c58c22ea08c P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.734459 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:47.750698 9469 consensus_peers.cc:485] T 21e68127f4f94413b1be984a87bec6a8 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:47.756460 6867 consensus_peers.cc:485] T 1b3ddfa17abc4ad2a64d8015724b4ae4 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.759536 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:47.765913 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:47.791414 9735 consensus_peers.cc:485] T 58548735201648fa805fdf6af390c088 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.811463 9467 consensus_peers.cc:485] T f217c08046f64be9ac120896c9a0c559 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:47.824224 9469 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.936450 9865 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.951890 6867 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.964056 7127 consensus_peers.cc:485] T 847ef240dcc64fb794ffcc06e483dc08 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:47.986918 6865 consensus_peers.cc:485] T f2f54c0c48b848ed898534efef55fb62 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.055707 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:48.121469 7125 consensus_peers.cc:485] T b97ddea52b624c668cb0b28ab0f38196 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:48.127981 6734 consensus_peers.cc:485] T 2d5781b2310f4814aaebdcb715b7ddac P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:48.285792 6736 consensus_peers.cc:485] T c59daa2d12f446899c7bb9970e1694da P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:48.290720 9735 consensus_peers.cc:485] T 1c727d4554464442a6a81a8ad4996c84 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.292883 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:48.321430 9469 consensus_peers.cc:485] T 5895d998bfb3439386fcbc2c74f8d9a6 P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:48.328394 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:48.329936 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:48.344090 7125 consensus_peers.cc:485] T 7e5410b22c57427d96a32fc9957cea8e P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:48.430966 9863 consensus_peers.cc:485] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:48.439013 9467 consensus_peers.cc:485] T e14af3987fbc44e896615d5229d815d7 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.451505 6865 consensus_peers.cc:485] T b3e2d7664e6c45d69f4f8b20dcf0e4d0 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:48.457552 9469 consensus_peers.cc:485] T 62a55d0a1d7c4d7aba6635de1994e60b P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.469600 6736 consensus_peers.cc:485] T 3b03512775d74ec9883ae6404eae58a3 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.485754 9467 consensus_peers.cc:485] T 7255e647ced644ed9b76937e7808a147 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:48.530921 9467 consensus_peers.cc:485] T 6e0bb82edac2464b9d618c1c80807fb2 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.540062 9863 consensus_peers.cc:485] T 3e5ba4e0e3044c1f85816ed69abc9386 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:15:48.587893 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:48.629165 9863 consensus_peers.cc:485] T 1461e0716aee4f74acefc639cf79eaf0 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.680920 9735 consensus_peers.cc:485] T 89ff0991a4e245609405ff945388417b P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.755824 9863 consensus_peers.cc:485] T df05d420e5cd426e88cd33de16a707f6 P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:15:48.798506 9467 consensus_peers.cc:485] T 77d3f36de08d4f1986e42e86e1b95fbf P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.807425 6736 consensus_peers.cc:485] T 09a755b35d694a2b89ca1f58f0fa7976 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:48.825506 6865 consensus_peers.cc:485] T 20c4e7c22c904daebe97897320f6226e P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:48.853013 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:48.871110 9733 consensus_peers.cc:485] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.879966 6865 consensus_peers.cc:485] T f7d921b017694f30af3e575690de44f8 P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:48.907531 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:48.937870 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:48.954183 6736 consensus_peers.cc:485] T dcb23bc5b0a44da7989aa828826ece07 P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.960729 9865 consensus_peers.cc:485] T d4d6c0c0b6c249208e67400face0b9a6 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:48.998574 9865 consensus_peers.cc:485] T f47877cdf5f04676861b3dc4708f3920 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:49.016667 9733 consensus_peers.cc:485] T 0b18e1d58117421c82ba86c08cd1253b P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.067250 9469 consensus_peers.cc:485] T 10ad2f2ef32d4fc88cdd62bfeab97b8a P 5b7e36bf354f480c90859cb8867b59cf -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.070466 7125 consensus_peers.cc:485] T 8aaca37078ac4c4eb7f0bb6df0654860 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:15:49.082587 6865 consensus_peers.cc:485] T 9f30640fc81e4aeead38e3f8659c056a P 7186ed8510424fd597173eaf10c57b12 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.089479 9865 consensus_peers.cc:485] T a9e7abf8b1cb44459dd0b103b786ff21 P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:49.116660 7125 consensus_peers.cc:485] T 669d722d3eb1475dba616ed72cb775ac P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.166261 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:49.175050 9733 consensus_peers.cc:485] T 9e5922e2c6624846b64abf6df41a5601 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.237308 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:49.249413 9865 consensus_peers.cc:485] T f6f8dfcb870a4461b03415db9c33452e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.256964 7125 consensus_peers.cc:485] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.258308 7127 consensus_peers.cc:485] T 652f2bf72ec9462384afcf9854509767 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:49.262188 9865 consensus_peers.cc:485] T e431272976034209b2c40cb0e460e17a P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:49.284741 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:49.305601 9735 consensus_peers.cc:485] T 5dc7cefd974149148dcf856a50ea168c P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:49.317287 9863 consensus_peers.cc:485] T 2f8a6c04df3443c0af8e4f20fcd171ff P ea801601713a4d55aee0de9e3473e0bc -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:15:49.358484 29485 raft_consensus.cc:482] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 09723f236d8044538d920a37b5d9aa8c)
I20240503 14:15:49.358664 29485 raft_consensus.cc:504] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "09723f236d8044538d920a37b5d9aa8c" member_type: VOTER last_known_addr { host: "127.0.186.3" port: 44691 } } peers { permanent_uuid: "5b7e36bf354f480c90859cb8867b59cf" member_type: VOTER last_known_addr { host: "127.0.186.5" port: 36609 } attrs { promote: false } } peers { permanent_uuid: "f03dc37fe3234033bc665bed4f1b51f5" member_type: VOTER last_known_addr { host: "127.0.186.6" port: 33971 } attrs { promote: false } }
I20240503 14:15:49.359481 29485 leader_election.cc:288] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691), f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971)
W20240503 14:15:49.362272 9469 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:49.362556 9467 leader_election.cc:334] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
I20240503 14:15:49.362653 9467 leader_election.cc:302] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5b7e36bf354f480c90859cb8867b59cf; no voters: 09723f236d8044538d920a37b5d9aa8c, f03dc37fe3234033bc665bed4f1b51f5
I20240503 14:15:49.362964 29485 raft_consensus.cc:2726] T 1d3ec4f0833d4d489f774757312efb32 P 5b7e36bf354f480c90859cb8867b59cf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240503 14:15:49.381955 7125 consensus_peers.cc:485] T 7c5bfc2d9491467caa7854c4b4c3ddf6 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:15:49.467844 7125 consensus_peers.cc:485] T b8014635e8294062b1c1985a100307ff P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.490625 9865 consensus_peers.cc:485] T 01e27516acca49459176362caf73595e P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:49.499545 6736 consensus_peers.cc:485] T 8477be3143f64d83a4c68f260a66efbf P f1a669e28f014fe287932acf56bebc13 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.500927 6867 consensus_peers.cc:485] T ca522e5b44fd44289b4b161f27f71d20 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 14:15:49.505142 7127 consensus_peers.cc:462] T 122296b00eda44949b9064d0afdd6a16 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:49.531263 7127 consensus_peers.cc:485] T 1708b19d3c5c4d308f4358bd42182c80 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:49.539418 9467 consensus_peers.cc:485] T 32d91b6f4bb244c78bc6fef8b837f964 P 5b7e36bf354f480c90859cb8867b59cf -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 14:15:49.543231 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(34f1ed0cfa8948048b3762919481367c): perf score=0.033341
W20240503 14:15:49.591545 7127 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 14:15:49.592207 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: FlushMRSOp(34f1ed0cfa8948048b3762919481367c) complete. Timing: real 0.049s user 0.040s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":321,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":2168,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":807,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":423,"threads_started":2,"wal-append.queue_time_us":305}
I20240503 14:15:49.594182 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling UndoDeltaBlockGCOp(34f1ed0cfa8948048b3762919481367c): 4838 bytes on disk
I20240503 14:15:49.595101 9470 maintenance_manager.cc:643] P 5b7e36bf354f480c90859cb8867b59cf: UndoDeltaBlockGCOp(34f1ed0cfa8948048b3762919481367c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240503 14:15:49.599383 9733 consensus_peers.cc:485] T ef4166613a824863b6c006dafb3783d3 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.637756 9735 consensus_peers.cc:485] T 4d04b4d914bc485fa3f437ad0eb40cb5 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 14:15:49.649415 7125 consensus_peers.cc:462] T 964b1598fb17446dacfe2b5931df5d41 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111)
W20240503 14:15:49.687932 9733 consensus_peers.cc:485] T 616e17b094b74234b3c59bc13e655f95 P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.730748 7125 consensus_peers.cc:485] T 9b154f780fc742fa9b0efd609c307f95 P e1bf5fb7402c46829974e2f74cd88708 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 14:15:49.743834 9735 consensus_peers.cc:462] T d133b4518f1f4decba0ba32cd82e528e P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:49.768114 9735 consensus_peers.cc:485] T 13f41f236987487b8bca287c7ec1fdd3 P 3efdbd3506274dba846ae40404f0572c -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:15:49.804450 9976 maintenance_manager.cc:419] P ea801601713a4d55aee0de9e3473e0bc: Scheduling FlushMRSOp(4b4363edcc954769a3ab9c658e198c2d): perf score=0.033390
W20240503 14:15:49.806919 9865 consensus_peers.cc:462] T 69009d7fef744e66800ce1f07b4a62bf P ea801601713a4d55aee0de9e3473e0bc -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111)
W20240503 14:15:49.826076 9733 consensus_peers.cc:485] T 1de841c504fc4bd78ea1a6a32bd34aeb P 3efdbd3506274dba846ae40404f0572c -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn't send request to peer f03dc37fe3234033bc665bed4f1b51f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.6:33971: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 14:15:49.845816 6867 consensus_peers.cc:485] T f5738d4265f941519b90b07334d06af9 P 7186ed8510424fd597173eaf10c57b12 -> Peer 09723f236d8044538d920a37b5d9aa8c (127.0.186.3:44691): Couldn't send request to peer 09723f236d8044538d920a37b5d9aa8c. Status: Network error: Client connection negotiation failed: client connection to 127.0.186.3:44691: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 14:15:49.849316 9579 maintenance_manager.cc:419] P 5b7e36bf354f480c90859cb8867b59cf: Scheduling FlushMRSOp(c8848a40563c43468ac46f8c604cc578): perf score=0.033344
I20240503 14:15:49.869966 9866 maintenance_manager.cc:643] P ea801601713a4d55aee0de9e3473e0bc: FlushMRSOp(4b4363edcc954769a3ab9c658e198c2d) complete. Timing: real 0.065s user 0.029s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3693,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":13267,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":819,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":4064,"threads_started":2,"wal-append.queue_time_us":352}
W20240503 14:15:49.872262 6734 consensus_peers.cc:485] T 9a89b1b988df4ad8aa8f1311cfa35731 P f1a669e28f014fe287932acf56bebc13 -> Peer f03dc37fe3234033bc665bed4f1b51f5 (127.0.186.6:33971): Couldn'